roxlu

roxlu

Member Since 11 years ago

roxlu, Amsterdam

Experience Points
340
follower
Lessons Completed
35
follow
Lessons Completed
152
stars
Best Reply Awards
20
repos

45 contributions in the last year

Pinned
⚡ Experiments with NVDECODE
⚡ Filament is a real-time physically based rendering engine for Android, iOS, Windows, Linux, macOS and WASM/WebGL
⚡ Experimental code for this issue https://github.com/google/filament/issues/1921
⚡ Example how to compile Freetype + Harfbuzz
⚡ Example that shows how to use Filament with GLFW and my feature-get-texture-id branch to use Filament with a shared OpenGL context
⚡ Experimenting with ExoPlayer2, hardware decoding + shared GL texture.
Activity
May
20
1 day ago
started
started time in 13 hours ago
May
18
3 days ago
May
17
4 days ago
Activity icon
issue

roxlu issue intel/libva-utils

roxlu
roxlu

Use of VAConfigAttribEncMaxRefFrames in h264encode.c

I'm looking into the h264encode.c example where the VAConfigAttribEncMaxRefFrames attribute is used to retrieve the number of maximum reference frames. When this attribute is supported the value is not checked against the num_ref_frames variable. What happens if num_ref_frames is larger than the value in the VAConfigAttribEncMaxRefFrames attribute? The num_ref_frames is used in the SPS.

May
7
2 weeks ago
started
started time in 2 weeks ago
May
2
2 weeks ago
started
started time in 2 weeks ago
started
started time in 2 weeks ago
started
started time in 2 weeks ago
Apr
29
3 weeks ago
started
started time in 3 weeks ago
Apr
23
4 weeks ago
started
started time in 4 weeks ago
started
started time in 4 weeks ago
Apr
22
4 weeks ago
started
started time in 4 weeks ago
Apr
19
1 month ago
Activity icon
issue

roxlu issue intel/libva

roxlu
roxlu

Question: How to extract SPS and PPS

After looking at several implemenations (GStreamer and FFmpeg) I've noticed that both these libraries create the SPS and PPS themself. However I'm sure the libva library has support to generate the SPS and PPS and I'm wondering how to extract these?

Apr
12
1 month ago
started
started time in 1 month ago
Mar
23
1 month ago
started
started time in 1 month ago
started
started time in 1 month ago
Mar
19
2 months ago
started
started time in 2 months ago
started
started time in 2 months ago
Mar
17
2 months ago
Activity icon
created branch

roxlu in roxlu/plugin-system create branch master

createdAt 2 months ago
Activity icon
created repository
createdAt 2 months ago
Mar
10
2 months ago
started
started time in 2 months ago
started
started time in 2 months ago
Mar
9
2 months ago
Activity icon
issue

roxlu issue comment awslabs/amazon-kinesis-video-streams-webrtc-sdk-c

roxlu
roxlu

lws connect() error errno=113

Occasionally, a connect() error occurs in lws. Is this related to the KVS server? Or is it a temporary network problem? Can I try again on state machine?

** Logging ** 7:38.646 Libwebsockets version: 4.0.22 v4.0.20-4-gb9cfc0c5 7:38.664 Initial logging level 4095 7:38.670 IPV6 not compiled in 7:38.673 LWS_DEF_HEADER_LEN : 4096 7:38.674 LWS_MAX_SMP : 1 7:38.680 SYSTEM_RANDOM_FILEPATH: '/dev/urandom' 7:38.682 HTTP2 support : not configured 7:38.683 [KVS] Using trickleICE by default, useTurn = TRUE 7:38.685 [KVS] set log level to 1 7:38.685 [KVS] Created signaling channel 328000373 7:38.725 [KVS] KVS WebRTC initialization completed successfully 7:38.731 KVS_INF createSignalingClientSync: Creating Signaling Client Sync 7:38.747 [lws lws_plat_init:137] mem: platform fd map: 256 bytes 7:38.749 [lws lws_plat_init:138] mem: max_fds 64 7:38.754 KVS_VER lwsHttpCallbackRoutine: HTTPS callback with reason 21 7:38.756 KVS_VER signalingClientStateChanged: Signaling client state changed to 1 - 'New' 7:38.759 KVS_DBG stepStateMachine: State Machine - Current state: 0x0000000000000001, Next state: 0x0000000000000002 7:38.782 KVS_VER signalingClientStateChanged: Signaling client state changed to 2 - 'Get Security Credentials' 7:38.786 KVS_DBG stepStateMachine: State Machine - Current state: 0x0000000000000002, Next state: 0x0000000000000004 7:38.790 KVS_VER signalingClientStateChanged: Signaling client state changed to 3 - 'Describe Channel' 7:38.815 KVS_VER lwsCompleteSync: Perform secure synchronous call for URL: https://kinesisvideo.ap-northeast-2.amazonaws.com/describeSignalingChannel 7:38.820 KVS_VER lwsHttpCallbackRoutine: HTTPS callback with reason 27 7:38.826 KVS_VER lwsHttpCallbackRoutine: HTTPS callback with reason 85 7:38.956 KVS_VER lwsHttpCallbackRoutine: HTTPS callback with reason 29 7:39.147 KVS_VER lwsHttpCallbackRoutine: HTTPS callback with reason 24 7:39.160 KVS_DBG lwsHttpCallbackRoutine: Client append handshake header 7:39.163 KVS_VER lwsHttpCallbackRoutine: Appending header - Authorization: AWS4-HMAC-SHA256 Credential=XXXXXXXXXXXXXXXXXXXX/20220223/ap-northeast-2/kinesisvideo/aws4_request, SignedHeaders=host;user-agent;x-amz-date, Signature=644b2b474c75d015ee0e4c28c9896db693fbab06041682b01b099f8cce594f12 7:39.184 KVS_VER lwsHttpCallbackRoutine: Appending header - content-length: 31 7:39.187 KVS_VER lwsHttpCallbackRoutine: Appending header - content-type: application/json 7:39.190 KVS_VER lwsHttpCallbackRoutine: Appending header - user-agent: AWS-SDK-KVS/3.0.0 arm-gcc iCatOS V37 7:39.257 KVS_VER lwsHttpCallbackRoutine: Appending header - X-Amz-Date: 20220223T054339Z 7:39.260 KVS_VER lwsHttpCallbackRoutine: HTTPS callback with reason 19 7:39.278 KVS_VER lwsHttpCallbackRoutine: HTTPS callback with reason 31 7:39.282 KVS_VER lwsHttpCallbackRoutine: HTTPS callback with reason 57 7:39.285 KVS_DBG lwsHttpCallbackRoutine: Sending the body { 7:39.287 "ChannelName": "328000373" 7:39.288 }, size 31 7:39.314 KVS_VER lwsHttpCallbackRoutine: HTTPS callback with reason 2 7:39.349 KVS_VER lwsHttpCallbackRoutine: HTTPS callback with reason 44 7:39.352 KVS_DBG lwsHttpCallbackRoutine: Connected with server response: 200 7:39.354 KVS_VER lwsHttpCallbackRoutine: HTTPS callback with reason 46 7:39.356 KVS_DBG lwsHttpCallbackRoutine: Received client http 7:39.359 KVS_VER lwsHttpCallbackRoutine: HTTPS callback with reason 48 7:39.361 KVS_DBG lwsHttpCallbackRoutine: Received client http read: 341 bytes 7:39.392 KVS_VER lwsHttpCallbackRoutine: HTTPS callback with reason 47 7:39.397 KVS_DBG lwsHttpCallbackRoutine: Http client completed 7:39.400 KVS_VER lwsHttpCallbackRoutine: HTTPS callback with reason 45 7:39.403 KVS_DBG lwsHttpCallbackRoutine: Client http closed 7:39.405 KVS_VER lwsHttpCallbackRoutine: HTTPS callback with reason 30 7:39.618 KVS_DBG stepStateMachine: State Machine - Current state: 0x0000000000000004, Next state: 0x0000000000000010 7:39.624 KVS_VER signalingClientStateChanged: Signaling client state changed to 5 - 'Get Channel Endpoint' 7:39.649 KVS_VER lwsCompleteSync: Perform secure synchronous call for URL: https://kinesisvideo.ap-northeast-2.amazonaws.com/getSignalingChannelEndpoint 7:39.663 KVS_VER lwsHttpCallbackRoutine: HTTPS callback with reason 85 7:55.304 lws_client_connect_3_connect: connect says errno: 113 7:55.321 lws_client_connect_3_connect: Connect failed: 13.209.140.207 port 443 7:55.325 KVS_VER lwsHttpCallbackRoutine: HTTPS callback with reason 30 7:55.331 KVS_DBG getChannelEndpoint: getChannelEndpointLws() 5d000015 7:55.346 KVS_DBG executeGetEndpointSignalingState: getChannelEndpoint() 0x5d000015 7:55.349 KVS_ERR createSignalingSync(179): operation returned status code: 0x0000000f 7:55.352 KVS_VER lwsHttpCallbackRoutine: HTTPS callback with reason 28 7:55.355 [KVS] createSignalingClientSync(): operation returned status code: 0x0000000f 7:55.358 [KVS] Terminated with status code 0x0000000f 7:55.359 [KVS] Cleaning up.... 7:55.360 KVS_VER signalingClientGetMetrics: Signaling Client Get Metrics 7:55.371 [KVS] signalingClientGetMetrics() operation returned status code: 0x00000001KVS_INF freeSignalingClient: Freeing Signaling Client

7:55.701 [KVS] Cleanup done

** Any design considerations/constraints **

Thanks.

roxlu
roxlu

@disa6302 I'm experiencing a similar issue that is easily reproduced with the v1.7.1 tag on Arch Linux.

Start the sample:

./kvsWebrtcClientMasterGstSample roxlu video-only testsrc

When the sample is running I disable my interface using:

sudo ip link set enp5s0 down

Then after a couple of seconds I bring up my network again:

udo ip link set enp5s0 up

And you keep receiving the messages shown below and you can't open a new connection with the channel anymore. :


[KVS GStreamer Master] Using trickleICE by default
[KVS GStreamer Master] Created signaling channel roxlu
[KVS Gstreamer Master] Finished initializing GStreamer
[KVS Gstreamer Master] Streaming video only
[KVS GStreamer Master] streaming type video-only[KVS GStreamer Master] KVS WebRTC initialization completed successfully
[2022/03/09 14:13:31:8310] N: LWS: 4.2.1-v4.2.2, loglevel 7
[2022/03/09 14:13:31:8310] N: NET CLI H1 H2 WS ConMon IPv6-absent
[2022/03/09 14:13:31:8350] N:  ++ [wsi|0|pipe] (1)
[2022/03/09 14:13:31:8374] N:  ++ [vh|0|netlink] (1)
[2022/03/09 14:13:31:8375] N:  ++ [vh|1|default||-1] (2)
[KVS GStreamer Master] Signaling client created successfully
[2022/03/09 14:13:31:8378] N:  ++ [wsicli|0|POST/h1/kinesisvideo.eu-west-2.amazonaws.com] (1)
[2022/03/09 14:13:31:9968] N:  -- [wsicli|0|POST/h1/kinesisvideo.eu-west-2.amazonaws.com] (0) 159.033ms
[2022/03/09 14:13:31:9969] N:  ++ [wsicli|1|POST/h1/kinesisvideo.eu-west-2.amazonaws.com] (1)
[2022/03/09 14:13:32:1495] N:  -- [wsicli|1|POST/h1/kinesisvideo.eu-west-2.amazonaws.com] (0) 152.658ms
[2022/03/09 14:13:32:1497] N:  ++ [wsicli|2|POST/h1/r-d29a6cb3.kinesisvideo.eu-west-2.amazonaws.] (1)
[2022/03/09 14:13:32:4164] N:  -- [wsicli|2|POST/h1/r-d29a6cb3.kinesisvideo.eu-west-2.amazonaws.] (0) 266.716ms
[2022/03/09 14:13:32:4165] N:  ++ [wsicli|3|WS/h1/m-978ce4ad.kinesisvideo.eu-west-2.amazonaws.co] (1)
[KVS GStreamer Master] Signaling client connection to socket established
[KVS Gstreamer Master] Beginning streaming...check the stream over channel roxlu
2022-03-09 13:14:59 ERROR   iceAgentAddRemoteCandidate(): operation returned status code: 0x5a000003
2022-03-09 13:14:59 ERROR   iceAgentAddRemoteCandidate(): operation returned status code: 0x5a000003
2022-03-09 13:14:59 ERROR   iceAgentAddRemoteCandidate(): operation returned status code: 0x5a000003
2022-03-09 13:14:59 ERROR   iceAgentAddRemoteCandidate(): operation returned status code: 0x5a000003
2022-03-09 13:14:59 WARN    handleStunPacket(): Cannot find candidate pair with local candidate 2001:1c00:2404:3100:a1b4:c33a:11c5:a0a7 and remote candidate 2001:1c00:2404:3100:a1b4:c33a:11c5:a0a7. Dropping STUN binding success response
[2022/03/09 14:15:03:7371] N: rops_handle_POLLIN_netlink: DELADDR
[2022/03/09 14:15:03:7371] N: _lws_route_check_wsi: [wsicli|3|WS/h1/m-978ce4ad.kinesisvideo.eu-west-2.amazonaws.co]: dest route gone
[2022/03/09 14:15:03:7372] N:  -- [wsicli|3|WS/h1/m-978ce4ad.kinesisvideo.eu-west-2.amazonaws.co] (0) 1.522min
[2022/03/09 14:15:03:9173] N:  ++ [wsicli|4|POST/h1/r-d29a6cb3.kinesisvideo.eu-west-2.amazonaws.] (1)
[2022/03/09 14:15:03:9180] N: lws_client_connect_3_connect: dns lookup failed -3
[2022/03/09 14:15:04:9197] N: lws_client_connect_3_connect: dns lookup failed -3
[2022/03/09 14:15:05:9213] N: lws_client_connect_3_connect: dns lookup failed -3
[2022/03/09 14:15:06:9215] N: lws_client_connect_3_connect: dns lookup failed -3
[2022/03/09 14:15:07:9227] N: lws_client_connect_3_connect: dns lookup failed -3
[2022/03/09 14:15:08:9230] N: lws_client_connect_3_connect: dns lookup failed -3
[2022/03/09 14:15:09:9234] N: lws_client_connect_3_connect: dns lookup failed -3
[2022/03/09 14:15:10:9244] N: lws_client_connect_3_connect: dns lookup failed -3
[2022/03/09 14:15:11:9250] N: lws_client_connect_3_connect: dns lookup failed -3
[2022/03/09 14:15:12:9262] N: lws_client_connect_3_connect: dns lookup failed -3
2022-03-09 13:15:13 WARN    lwsHttpCallbackRoutine(): Client connection failed. Connection error string: Closed before conn
[2022/03/09 14:15:13:9180] N:  -- [wsicli|4|POST/h1/r-d29a6cb3.kinesisvideo.eu-west-2.amazonaws.] (0) 10.000s
2022-03-09 13:15:17 ERROR   getIceConfigLws(): Call Failed with Status:  0x5d000034
[2022/03/09 14:15:17:9183] N:  ++ [wsicli|5|POST/h1/r-d29a6cb3.kinesisvideo.eu-west-2.amazonaws.] (1)
[2022/03/09 14:15:17:9184] N: lws_client_connect_3_connect: dns lookup failed -3
[2022/03/09 14:15:18:9192] N: lws_client_connect_3_connect: dns lookup failed -3
[2022/03/09 14:15:19:9204] N: lws_client_connect_3_connect: dns lookup failed -3
[2022/03/09 14:15:20:9463] N: lws_sort_dns: 3.9.32.85 has no route out
[2022/03/09 14:15:20:9463] N: lws_sort_dns: 18.133.18.180 has no route out
[2022/03/09 14:15:20:9463] N: lws_sort_dns: 35.179.98.11 has no route out
[2022/03/09 14:15:20:9463] N: lws_sort_dns_dump: empty
2022-03-09 13:15:20 WARN    lwsHttpCallbackRoutine(): Client connection failed. Connection error string: closed before established
[2022/03/09 14:15:20:9464] N:  -- [wsicli|5|POST/h1/r-d29a6cb3.kinesisvideo.eu-west-2.amazonaws.] (0) 3.028s
[2022/03/09 14:15:20:9464] N: lws_client_dns_retry_timeout: DNS lookup failed
2022-03-09 13:15:24 ERROR   getIceConfigLws(): Call Failed with Status:  0x5d000034
2022-03-09 13:15:24 WARN    signalingClientError(): Signaling client generated an error 0x5d000030 - 'Failed to reconnect with status code 0x0000000f.'
[2022/03/09 14:15:24:1661] N:  ++ [wsicli|6|POST/h1/r-d29a6cb3.kinesisvideo.eu-west-2.amazonaws.] (1)
[2022/03/09 14:15:24:2050] N: lws_sort_dns: 3.9.32.85 has no route out
[2022/03/09 14:15:24:2050] N: lws_sort_dns: 35.179.98.11 has no route out
[2022/03/09 14:15:24:2050] N: lws_sort_dns: 18.133.18.180 has no route out
[2022/03/09 14:15:24:2050] N: lws_sort_dns_dump: empty
[2022/03/09 14:15:24:2050] N:  -- [wsicli|6|POST/h1/r-d29a6cb3.kinesisvideo.eu-west-2.amazonaws.] (0) 38.927ms
2022-03-09 13:15:24 ERROR   getIceConfigLws(): Call Failed with Status:  0x5d000015
[2022/03/09 14:15:24:2050] N:  ++ [wsicli|7|POST/h1/r-d29a6cb3.kinesisvideo.eu-west-2.amazonaws.] (1)
[2022/03/09 14:15:24:2306] N: lws_sort_dns: 35.179.7.193 has no route out
[2022/03/09 14:15:24:2306] N: lws_sort_dns: 18.171.0.4 has no route out
[2022/03/09 14:15:24:2306] N: lws_sort_dns: 18.135.169.155 has no route out
[2022/03/09 14:15:24:2306] N: lws_sort_dns_dump: empty
[2022/03/09 14:15:24:2306] N:  -- [wsicli|7|POST/h1/r-d29a6cb3.kinesisvideo.eu-west-2.amazonaws.] (0) 25.591ms
2022-03-09 13:15:24 ERROR   getIceConfigLws(): Call Failed with Status:  0x5d000015
2022-03-09 13:15:24 ERROR   signalingFetchSync(): operation returned status code: 0x5d000006
2022-03-09 13:15:24 ERROR   signalingClientFetchSync(): Create Signaling Sync API returned [0x5d000006]  7

[2022/03/09 14:15:25:8996] N:  ++ [wsicli|8|POST/h1/r-d29a6cb3.kinesisvideo.eu-west-2.amazonaws.] (1)
[2022/03/09 14:15:25:9210] N: lws_sort_dns: 3.9.32.85 has no route out
[2022/03/09 14:15:25:9210] N: lws_sort_dns: 35.179.98.11 has no route out
[2022/03/09 14:15:25:9210] N: lws_sort_dns: 18.133.18.180 has no route out
[2022/03/09 14:15:25:9210] N: lws_sort_dns_dump: empty
[2022/03/09 14:15:25:9211] N:  -- [wsicli|8|POST/h1/r-d29a6cb3.kinesisvideo.eu-west-2.amazonaws.] (0) 21.420ms
2022-03-09 13:15:25 ERROR   getIceConfigLws(): Call Failed with Status:  0x5d000015
[2022/03/09 14:15:25:9211] N:  ++ [wsicli|9|POST/h1/r-d29a6cb3.kinesisvideo.eu-west-2.amazonaws.] (1)
[2022/03/09 14:15:25:9460] N: lws_sort_dns: 18.133.18.180 has no route out
[2022/03/09 14:15:25:9460] N: lws_sort_dns: 3.9.32.85 has no route out
[2022/03/09 14:15:25:9460] N: lws_sort_dns: 35.179.98.11 has no route out
[2022/03/09 14:15:25:9461] N: lws_sort_dns_dump: empty
[2022/03/09 14:15:25:9461] N:  -- [wsicli|9|POST/h1/r-d29a6cb3.kinesisvideo.eu-west-2.amazonaws.] (0) 24.969ms
2022-03-09 13:15:25 ERROR   getIceConfigLws(): Call Failed with Status:  0x5d000015
2022-03-09 13:15:25 ERROR   signalingFetchSync(): operation returned status code: 0x5d000006
2022-03-09 13:15:25 ERROR   signalingClientFetchSync(): Create Signaling Sync API returned [0x5d000006]  6

[2022/03/09 14:15:28:2717] N:  ++ [wsicli|a|POST/h1/r-d29a6cb3.kinesisvideo.eu-west-2.amazonaws.] (1)
[2022/03/09 14:15:28:2929] N: lws_sort_dns: 18.171.0.4 has no route out
[2022/03/09 14:15:28:2929] N: lws_sort_dns: 18.135.169.155 has no route out
[2022/03/09 14:15:28:2929] N: lws_sort_dns: 35.179.7.193 has no route out
[2022/03/09 14:15:28:2929] N: lws_sort_dns_dump: empty
[2022/03/09 14:15:28:2929] N:  -- [wsicli|a|POST/h1/r-d29a6cb3.kinesisvideo.eu-west-2.amazonaws.] (0) 21.271ms
2022-03-09 13:15:28 ERROR   getIceConfigLws(): Call Failed with Status:  0x5d000015
[2022/03/09 14:15:28:2930] N:  ++ [wsicli|b|POST/h1/r-d29a6cb3.kinesisvideo.eu-west-2.amazonaws.] (1)
[2022/03/09 14:15:28:3105] N: lws_sort_dns: 35.179.7.193 has no route out
[2022/03/09 14:15:28:3105] N: lws_sort_dns: 18.171.0.4 has no route out
[2022/03/09 14:15:28:3105] N: lws_sort_dns: 18.135.169.155 has no route out
[2022/03/09 14:15:28:3105] N: lws_sort_dns_dump: empty
[2022/03/09 14:15:28:3106] N:  -- [wsicli|b|POST/h1/r-d29a6cb3.kinesisvideo.eu-west-2.amazonaws.] (0) 17.574ms
2022-03-09 13:15:28 ERROR   getIceConfigLws(): Call Failed with Status:  0x5d000015
2022-03-09 13:15:28 ERROR   signalingFetchSync(): operation returned status code: 0x5d000006
2022-03-09 13:15:28 ERROR   signalingClientFetchSync(): Create Signaling Sync API returned [0x5d000006]  5

[2022/03/09 14:15:32:4778] N:  ++ [wsicli|c|POST/h1/r-d29a6cb3.kinesisvideo.eu-west-2.amazonaws.] (1)
[2022/03/09 14:15:32:5034] N: lws_sort_dns: 35.179.7.193 has no route out
[2022/03/09 14:15:32:5034] N: lws_sort_dns: 18.135.169.155 has no route out
[2022/03/09 14:15:32:5034] N: lws_sort_dns: 18.171.0.4 has no route out
[2022/03/09 14:15:32:5034] N: lws_sort_dns_dump: empty
[2022/03/09 14:15:32:5034] N:  -- [wsicli|c|POST/h1/r-d29a6cb3.kinesisvideo.eu-west-2.amazonaws.] (0) 25.654ms
2022-03-09 13:15:32 ERROR   getIceConfigLws(): Call Failed with Status:  0x5d000015
[2022/03/09 14:15:32:5035] N:  ++ [wsicli|d|POST/h1/r-d29a6cb3.kinesisvideo.eu-west-2.amazonaws.] (1)
[2022/03/09 14:15:32:5425] N: lws_sort_dns: 18.133.18.180 has no route out
[2022/03/09 14:15:32:5425] N: lws_sort_dns: 35.179.98.11 has no route out
[2022/03/09 14:15:32:5425] N: lws_sort_dns: 3.9.32.85 has no route out
[2022/03/09 14:15:32:5425] N: lws_sort_dns_dump: empty
[2022/03/09 14:15:32:5425] N:  -- [wsicli|d|POST/h1/r-d29a6cb3.kinesisvideo.eu-west-2.amazonaws.] (0) 39.078ms
2022-03-09 13:15:32 ERROR   getIceConfigLws(): Call Failed with Status:  0x5d000015
2022-03-09 13:15:32 ERROR   signalingFetchSync(): operation returned status code: 0x5d000006
2022-03-09 13:15:32 ERROR   signalingClientFetchSync(): Create Signaling Sync API returned [0x5d000006]  4

[2022/03/09 14:15:40:8698] N:  ++ [wsicli|e|POST/h1/r-d29a6cb3.kinesisvideo.eu-west-2.amazonaws.] (1)
[2022/03/09 14:15:40:8959] N: lws_sort_dns: 18.135.212.14 has no route out
[2022/03/09 14:15:40:8959] N: lws_sort_dns: 3.11.40.34 has no route out
[2022/03/09 14:15:40:8959] N: lws_sort_dns: 18.132.158.30 has no route out
[2022/03/09 14:15:40:8959] N: lws_sort_dns_dump: empty
[2022/03/09 14:15:40:8959] N:  -- [wsicli|e|POST/h1/r-d29a6cb3.kinesisvideo.eu-west-2.amazonaws.] (0) 26.169ms
2022-03-09 13:15:40 ERROR   getIceConfigLws(): Call Failed with Status:  0x5d000015
[2022/03/09 14:15:40:8960] N:  ++ [wsicli|f|POST/h1/r-d29a6cb3.kinesisvideo.eu-west-2.amazonaws.] (1)
[2022/03/09 14:15:40:9195] N: lws_sort_dns: 18.133.18.180 has no route out
[2022/03/09 14:15:40:9195] N: lws_sort_dns: 3.9.32.85 has no route out
[2022/03/09 14:15:40:9195] N: lws_sort_dns: 35.179.98.11 has no route out
[2022/03/09 14:15:40:9195] N: lws_sort_dns_dump: empty
[2022/03/09 14:15:40:9195] N:  -- [wsicli|f|POST/h1/r-d29a6cb3.kinesisvideo.eu-west-2.amazonaws.] (0) 23.505ms
2022-03-09 13:15:40 ERROR   getIceConfigLws(): Call Failed with Status:  0x5d000015
2022-03-09 13:15:40 ERROR   signalingFetchSync(): operation returned status code: 0x5d000006
2022-03-09 13:15:40 ERROR   signalingClientFetchSync(): Create Signaling Sync API returned [0x5d000006]  3

[2022/03/09 14:16:01:3200] N:  ++ [wsicli|10|POST/h1/r-d29a6cb3.kinesisvideo.eu-west-2.amazonaws] (1)
[2022/03/09 14:16:01:3389] N: lws_sort_dns: 18.133.18.180 has no route out
[2022/03/09 14:16:01:3389] N: lws_sort_dns: 35.179.98.11 has no route out
[2022/03/09 14:16:01:3390] N: lws_sort_dns: 3.9.32.85 has no route out
[2022/03/09 14:16:01:3390] N: lws_sort_dns_dump: empty
[2022/03/09 14:16:01:3390] N:  -- [wsicli|10|POST/h1/r-d29a6cb3.kinesisvideo.eu-west-2.amazonaws] (0) 18.972ms
2022-03-09 13:16:01 ERROR   getIceConfigLws(): Call Failed with Status:  0x5d000015
[2022/03/09 14:16:01:3390] N:  ++ [wsicli|11|POST/h1/r-d29a6cb3.kinesisvideo.eu-west-2.amazonaws] (1)
[2022/03/09 14:16:01:3632] N: lws_sort_dns: 18.135.169.155 has no route out
[2022/03/09 14:16:01:3632] N: lws_sort_dns: 18.171.0.4 has no route out
[2022/03/09 14:16:01:3632] N: lws_sort_dns: 35.179.7.193 has no route out
[2022/03/09 14:16:01:3632] N: lws_sort_dns_dump: empty
[2022/03/09 14:16:01:3632] N:  -- [wsicli|11|POST/h1/r-d29a6cb3.kinesisvideo.eu-west-2.amazonaws] (0) 24.180ms
2022-03-09 13:16:01 ERROR   getIceConfigLws(): Call Failed with Status:  0x5d000015
2022-03-09 13:16:01 ERROR   signalingFetchSync(): operation returned status code: 0x5d000006
2022-03-09 13:16:01 ERROR   signalingClientFetchSync(): Create Signaling Sync API returned [0x5d000006]  2

[2022/03/09 14:16:22:1501] N:  ++ [wsicli|12|POST/h1/r-d29a6cb3.kinesisvideo.eu-west-2.amazonaws] (1)
[2022/03/09 14:16:22:1735] N: lws_sort_dns: 3.9.32.85 has no route out
[2022/03/09 14:16:22:1735] N: lws_sort_dns: 18.133.18.180 has no route out
[2022/03/09 14:16:22:1735] N: lws_sort_dns: 35.179.98.11 has no route out
[2022/03/09 14:16:22:1735] N: lws_sort_dns_dump: empty
[2022/03/09 14:16:22:1735] N:  -- [wsicli|12|POST/h1/r-d29a6cb3.kinesisvideo.eu-west-2.amazonaws] (0) 23.426ms
2022-03-09 13:16:22 ERROR   getIceConfigLws(): Call Failed with Status:  0x5d000015
[2022/03/09 14:16:22:1736] N:  ++ [wsicli|13|POST/h1/r-d29a6cb3.kinesisvideo.eu-west-2.amazonaws] (1)
[2022/03/09 14:16:22:2098] N: lws_sort_dns: 35.179.7.193 has no route out
[2022/03/09 14:16:22:2098] N: lws_sort_dns: 18.171.0.4 has no route out
[2022/03/09 14:16:22:2098] N: lws_sort_dns: 18.135.169.155 has no route out
[2022/03/09 14:16:22:2098] N: lws_sort_dns_dump: empty
[2022/03/09 14:16:22:2099] N:  -- [wsicli|13|POST/h1/r-d29a6cb3.kinesisvideo.eu-west-2.amazonaws] (0) 36.270ms
2022-03-09 13:16:22 ERROR   getIceConfigLws(): Call Failed with Status:  0x5d000015
2022-03-09 13:16:22 ERROR   signalingFetchSync(): operation returned status code: 0x5d000006
2022-03-09 13:16:22 ERROR   signalingClientFetchSync(): Create Signaling Sync API returned [0x5d000006]  1
Activity icon
issue

roxlu issue awslabs/amazon-kinesis-video-streams-producer-c

roxlu
roxlu

[QUESTION] Thread safety: create `STREAM_HANDLE` in thread A and call `putKinesisVideoFrame()` from thread B.

I've been running into a situation where my application, that uses this KVS Producer SDK, suddenly hangs/freezes.

The interesting thing is that this only happens when I test on a RPI and not on my main intel i7 development PC. After inspecting the state with GDB I noticed something which looks like two threads wait on the same lock, but I'm not yet 100% sure. I'm creating a STREAM_HANDLE instance in thread A, then I start a thread which collects RTP data and then calls putKinesisVideoFrame(). Once the stream has started the STREAM_HANDLE is not used by any other thread from my app. As there are no data-races, at least not that I'm aware off, this shouldn't be a problem. But to be sure I synchronise access to my STREAM_HANDLE using a mutex.

I've collected some data and I'm wondering if my app "freezes" on these locks. When you look below at Thread 4, KVS-1 and the call to lockMutexFn(pKinesisVideoClient->clientCallbacks.customData, ...) and Thread 9, KVS-2 you can see that they both try to lock the same mutex.

--

Run 1

The last output looks like:

2022-02-18 20:48:02 DEBUG   postReadCallback(): Wrote 843 bytes to Kinesis Video. Upload stream handle: 0                                                                 

[21:48:02] debug [   connection.c: 293] |> Start to read.
[21:48:02] debug [   connection.c: 299] <| Read 26873 bytes.
[21:48:02] debug [       pusher.c: 464] |> calling `putKinesisVideoFrame()` with 26843 bytes.                                                                             

2022-02-18 20:48:02 DEBUG   postReadCallback(): Wrote 708 bytes to Kinesis Video. Upload stream handle: 0                                                                 
2022-02-18 20:48:02 DEBUG   postReadCallback(): Pausing CURL read for upload handle: 0                                                                                    
2022-02-18 20:48:37 WARN    curlCompleteSync(): curl perform failed for url https://s-0cb96ab8.kinesisvideo.eu-west-2.amazonaws.com/putMedia with result Timeout was reached: Operation too slow. Less than 30 bytes/sec transferred the last 30 seconds
2022-02-18 20:48:37 WARN    curlCompleteSync(): HTTP Error 0 : Response: (null)

Request URL: https://s-0cb96ab8.kinesisvideo.eu-west-2.amazonaws.com/putMedia
Request Headers:
    Authorization: AWS4-HMAC-SHA256 Credential=<cred>/20220218/eu-west-2/kinesisvideo/aws4_request, SignedHeaders=connection;host;transfer-encoding;user-agent;x-amz-date;x-amzn-fragment-acknowledgment-required;x-amzn-fragment-timecode-type;x-amzn-producer-start-timestamp;x-amzn-stream-name, Signature=<sig>
    connection: keep-alive
    content-type: application/json
    host: s-0cb96ab8.kinesisvideo.eu-west-2.amazonaws.com
    transfer-encoding: chunked
    user-agent: AWS-SDK-KVS/3.0.0 GCC/8.3.0 Linux/5.10.52-v7l+ armv7l
    X-Amz-Date: 20220218T204024Z
    x-amzn-fragment-acknowledgment-required: 1
    x-amzn-fragment-timecode-type: RELATIVE
    x-amzn-producer-start-timestamp: 1645216824.732
    x-amzn-stream-name: roxlu
    
2022-02-18 20:48:37 DEBUG   putStreamCurlHandler(): Network thread for Kinesis Video stream: roxlu with upload handle: 0 exited. http status: 0
2022-02-18 20:48:37 WARN    putStreamCurlHandler(): Stream with streamHandle 18446744072315065456 uploadHandle 0 has exited without triggering end-of-stream. Service call result: 599
2022-02-18 20:48:37 INFO    kinesisVideoStreamTerminated(): Stream 0xfffffffface1d470 terminated upload handle 0 with service call result 599.
2022-02-18 20:48:37 DEBUG   defaultStreamStateTransitionHook(): Stream state machine retry count: 0
2022-02-18 20:48:37 DEBUG   defaultStreamStateTransitionHook():
 
 KinesisVideoStream base result is [599]. Executing KVS retry handler of retry strategy type [1]
 
[Thread 0xa27fea90 (LWP 17414) exited]

Using GDB I can see there are several threads:

(gdb) info threads 
  Id   Target Id                                    Frame 
  1    Thread 0xae1e6920 (LWP 17402) "pusher-debug" 0xb5574a3c in __GI___pthread_timedjoin_ex (threadid=2910841488, [email protected]=0xbeffeaa8, [email protected]=0x0, [email protected]=true) at pthread_join_common.c:89
  2    Thread 0xae1a1a90 (LWP 17406) "pusher-debug" futex_wait_cancelable (private=0, expected=0, futex_word=0x3250b0 <glog+104>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  3    Thread 0xad7fea90 (LWP 17407) "pusher-debug" __GI___select (timeout=0xbeffea4a, exceptfds=0x0, writefds=0x0, readfds=0xad7fe3ac, nfds=5) at ../sysdeps/unix/sysv/linux/select.c:41
  4    Thread 0xa39fca90 (LWP 17408) "pusher-debug" __lll_lock_wait ([email protected]=0xace24980, private=0) at lowlevellock.c:46
* 9    Thread 0xa31fba90 (LWP 17413) "pusher-debug" __lll_lock_wait ([email protected]=0xace1d740, private=0) at lowlevellock.c:46
  • Thread 1: this main, which joins my server/application thread.
  • Thread 2: this is my logger thread; waits for a condition to signal
  • Thread 3: the server thread; accepts a connection on which is receives media packets

Thread 1, main:

(gdb) thread 1
[Switching to thread 1 (Thread 0xae1e6920 (LWP 17402))]
#0  0xb5574a3c in __GI___pthread_timedjoin_ex (threadid=2910841488, [email protected]=0xbeffeaa8, [email protected]=0x0, [email protected]=true) at pthread_join_common.c:89
89      pthread_join_common.c: No such file or directory.

(gdb) bt
#0  0xb5574a3c in __GI___pthread_timedjoin_ex (threadid=2910841488, [email protected]=0xbeffeaa8, [email protected]=0x0, [email protected]=true) at pthread_join_common.c:89
#1  0xb5581008 in thrd_join (thr=<optimized out>, res=0x0) at thrd_join.c:25
#2  0x0001a2c8 in rx_pusher_server_join (ctx=0x354130) at /<home>/webrtc-pusher/src/rx/pusher/server.c:187
#3  0x00015550 in main (argc=5, argv=0xbeffec74) at /<home>/webrtc-pusher/src/app/pusher.c:83

Thread 2, logger:

#0  0xb557a1a0 in futex_wait_cancelable (private=0, expected=0, futex_word=0x3250b0 <glog+104>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  0xb557a1a0 in __pthread_cond_wait_common (abstime=0x0, mutex=0x0, cond=0x325088 <glog+64>) at pthread_cond_wait.c:502
#2  0xb557a1a0 in __pthread_cond_wait (cond=0x325088 <glog+64>, mutex=0x0) at pthread_cond_wait.c:655
#3  0xb5581388 in cnd_wait (cond=<optimized out>, mutex=<optimized out>) at cnd_wait.c:24
#4  0x0001697c in rx_log_thread (user=0x325048 <glog>) at /<home>/src/log.c:665
#5  0xb5573494 in start_thread (arg=0xae1a1a90) at pthread_create.c:486
#6  0xb54f6578 in  () at ../sysdeps/unix/sysv/linux/arm/clone.S:73

Thread 3, server thread that accepts client connections:

(gdb) thread 3

[Switching to thread 3 (Thread 0xad7fea90 (LWP 17407))]
#0  __GI___select (timeout=0xbeffea4a, exceptfds=0x0, writefds=0x0, readfds=0xad7fe3ac, nfds=5) at ../sysdeps/unix/sysv/linux/select.c:41
41      ../sysdeps/unix/sysv/linux/select.c: No such file or directory.

(gdb) list

36      in ../sysdeps/unix/sysv/linux/select.c

(gdb) bt

#0  0xb54ef6ac in __GI___select (timeout=0xbeffea4a, exceptfds=0x0, writefds=0x0, readfds=0xad7fe3ac, nfds=5) at ../sysdeps/unix/sysv/linux/select.c:41
#1  0xb54ef6ac in __GI___select (nfds=5, readfds=0xad7fe3ac, writefds=0x0, exceptfds=0x0, timeout=0xad7fe42c) at ../sysdeps/unix/sysv/linux/select.c:37
#2  0x0001d4f8 in rx_socket_can_read (ctx=0x33c0e8, timeoutSec=3, timeoutMicroSec=500000, result=0xad7fe467 "\003\310\005ସ", <incomplete sequence \340\254>) at /<home>/webrtc-pusher/src/rx/socket.c:692
#3  0x0001a758 in server_thread_func (user=0x354130) at /<home>/webrtc-pusher/src/rx/pusher/server.c:317
#4  0xb5573494 in start_thread (arg=0xad7fea90) at pthread_create.c:486
#5  0xb54f6578 in  () at ../sysdeps/unix/sysv/linux/arm/clone.S:73
(gdb) f 2
#2  0x0001d4f8 in rx_socket_can_read (ctx=0x33c0e8, timeoutSec=3, timeoutMicroSec=500000, result=0xad7fe467 "\003\310\005ସ", <incomplete sequence \340\254>) at /<home>/webrtc-pusher/src/rx/socket.c:692
692       r = select(ctx->handle + 1, &readset, NULL, NULL, &timeout);

(gdb) list
687       timeout.tv_usec = timeoutMicroSec;
688       
689       FD_ZERO(&readset);
690       FD_SET(ctx->handle, &readset);
691
692       r = select(ctx->handle + 1, &readset, NULL, NULL, &timeout);
693       
694       if (0 == r) {
695         /* Nope, timed out. */
696         *result = RX_SOCK_TIMEOUT;

Thread 4, KVS-1

(gdb) thread 4
[Switching to thread 4 (Thread 0xa39fca90 (LWP 17408))]
#0  __lll_lock_wait ([email protected]=0xace24980, private=0) at lowlevellock.c:46
46      lowlevellock.c: No such file or directory.

(gdb) bt
#0  0xb557de98 in __lll_lock_wait ([email protected]=0xace24980, private=0) at lowlevellock.c:46
#1  0xb5575f58 in __GI___pthread_mutex_lock (mutex=0xace24980) at pthread_mutex_lock.c:115
#2  0x00063318 in defaultLockMutex (mutex=18446744072315095424) at /<home>/webrtc-pusher/build/build.debug/amazonproducer/src/amazonproducer/dependency/libkvspic/kvspic-src/src/utils/src/Mutex.c:195
#3  0x00043e9c in kinesisVideoStreamDefaultLockMutex (customData=18446744072314954400, mutex=18446744072315095424) at /<home>/webrtc-pusher/build/build.debug/amazonproducer/src/amazonproducer/dependency/libkvspic/kvspic-src/src/client/src/Callbacks.c:40
#4  0x000443c0 in checkIntermittentProducerCallback (timerId=0, currentTime=16452172820799116, customData=18446744072315038536) at /<home>/webrtc-pusher/build/build.debug/amazonproducer/src/amazonproducer/dependency/libkvspic/kvspic-src/src/client/src/Client.c:64
#5  0x00069a8c in timerQueueExecutor (args=0xace1d8b0) at /<home>/webrtc-pusher/build/build.debug/amazonproducer/src/amazonproducer/dependency/libkvspic/kvspic-src/src/utils/src/TimerQueue.c:560
#6  0xb5573494 in start_thread (arg=0xa39fca90) at pthread_create.c:486
#7  0xb54f6578 in  () at ../sysdeps/unix/sysv/linux/arm/clone.S:73

(gdb) f 3
#3  0x00043e9c in kinesisVideoStreamDefaultLockMutex (customData=18446744072314954400, mutex=18446744072315095424) at /<home>/webrtc-pusher/build/build.debug/amazonproducer/src/amazonproducer/dependency/libkvspic/kvspic-src/src/client/src/Callbacks.c:40
40          MUTEX_LOCK(mutex);
(gdb) list
35       * Default lock mutex functionality
36       */
37      VOID kinesisVideoStreamDefaultLockMutex(UINT64 customData, MUTEX mutex)
38      {
39          UNUSED_PARAM(customData);
40          MUTEX_LOCK(mutex);
41      }
42
43      /**

(gdb) f 4
#4  0x000443c0 in checkIntermittentProducerCallback (timerId=0, currentTime=16452172820799116, customData=18446744072315038536) at /<home>/webrtc-pusher/build/build.debug/amazonproducer/src/amazonproducer/dependency/libkvspic/kvspic-src/src/client/src/Client.c:64
64                          pKinesisVideoClient->clientCallbacks.lockMutexFn(pKinesisVideoClient->clientCallbacks.customData, pFrameOrderCoordinator->lock);
(gdb) list
59                      pCurrStream = pKinesisVideoClient->streams[i];
60                      pFrameOrderCoordinator = pCurrStream->pFrameOrderCoordinator;
61                      if (pCurrStream->streamInfo.streamCaps.frameOrderingMode != FRAME_ORDER_MODE_PASS_THROUGH) {
62                          // In the case that frameOrderingMode = FRAME_ORDER_MODE_PASS_THROUGH, pFrameOrderCoordinator is NULL
63                          // so we will segfault, so the check for the frameOrderingMode is important
64                          pKinesisVideoClient->clientCallbacks.lockMutexFn(pKinesisVideoClient->clientCallbacks.customData, pFrameOrderCoordinator->lock);
65                          frameOrderCoordinatorLocked = TRUE;
66                      }
67                      // Lock the Stream
68                      pKinesisVideoClient->clientCallbacks.lockMutexFn(pKinesisVideoClient->clientCallbacks.customData, pCurrStream->base.lock);

Thread 9, KVS-2

(gdb) thread 9
[Switching to thread 9 (Thread 0xa31fba90 (LWP 17413))]
#0  __lll_lock_wait ([email protected]=0xace1d740, private=0) at lowlevellock.c:46
46      lowlevellock.c: No such file or directory.

(gdb) bt
#0  0xb557de98 in __lll_lock_wait ([email protected]=0xace1d740, private=0) at lowlevellock.c:46
#1  0xb5575f58 in __GI___pthread_mutex_lock (mutex=0xace1d740) at pthread_mutex_lock.c:115
#2  0x00063318 in defaultLockMutex (mutex=18446744072315066176) at /<home>/webrtc-pusher/build/build.debug/amazonproducer/src/amazonproducer/dependency/libkvspic/kvspic-src/src/utils/src/Mutex.c:195
#3  0x00043e9c in kinesisVideoStreamDefaultLockMutex (customData=18446744072314954400, mutex=18446744072315066176) at /<home>/webrtc-pusher/build/build.debug/amazonproducer/src/amazonproducer/dependency/libkvspic/kvspic-src/src/client/src/Callbacks.c:40
#4  0x00045678 in getKinesisVideoMetrics (clientHandle=18446744072315038536, pKinesisVideoMetrics=0xa31fb028) at /<home>/webrtc-pusher/build/build.debug/amazonproducer/src/amazonproducer/dependency/libkvspic/kvspic-src/src/client/src/Client.c:459
#5  0x0004f56c in logStreamMetric (pKinesisVideoStream=0xace1e018) at /<home>/webrtc-pusher/build/build.debug/amazonproducer/src/amazonproducer/dependency/libkvspic/kvspic-src/src/client/src/Stream.c:674
#6  0x00050e50 in putFrame (pKinesisVideoStream=0xace1e018, pFrame=0xa1e1b488) at /<home>/webrtc-pusher/build/build.debug/amazonproducer/src/amazonproducer/dependency/libkvspic/kvspic-src/src/client/src/Stream.c:991
#7  0x0004c048 in putEarliestFrame (pKinesisVideoStream=0xace1e018) at /<home>/webrtc-pusher/build/build.debug/amazonproducer/src/amazonproducer/dependency/libkvspic/kvspic-src/src/client/src/FrameOrderCoordinator.c:233
#8  0x0004c4bc in frameOrderCoordinatorPutFrame (pKinesisVideoStream=0xace1e018, pUserFrame=0xa31fb3f8) at /<home>/webrtc-pusher/build/build.debug/amazonproducer/src/amazonproducer/dependency/libkvspic/kvspic-src/src/client/src/FrameOrderCoordinator.c:319
#9  0x00046d14 in putKinesisVideoFrame (streamHandle=18446744072315065456, pFrame=0xa31fb3f8) at /<home>/webrtc-pusher/build/build.debug/amazonproducer/src/amazonproducer/dependency/libkvspic/kvspic-src/src/client/src/Client.c:876
#10 0x00022478 in rx_pusher_send_frame (ctx=0xace00610, frame=0xa31fb3f8) at /<home>/webrtc-pusher/src/rx/pusher.c:465
#11 0x0001b9bc in connection_thread_func (user=0xace005c8) at /<home>/webrtc-pusher/src/rx/pusher/connection.c:354
#12 0xb5573494 in start_thread (arg=0xa31fba90) at pthread_create.c:486
#13 0xb54f6578 in  () at ../sysdeps/unix/sysv/linux/arm/clone.S:73

(gdb) f 2
#2  0x00063318 in defaultLockMutex (mutex=18446744072315066176) at /<home>/webrtc-pusher/build/build.debug/amazonproducer/src/amazonproducer/dependency/libkvspic/kvspic-src/src/utils/src/Mutex.c:195
195         pthread_mutex_lock((pthread_mutex_t*) mutex);
(gdb) list
190         return (MUTEX) pMutex;
191     }
192
193     VOID defaultLockMutex(MUTEX mutex)
194     {
195         pthread_mutex_lock((pthread_mutex_t*) mutex);
196     }
197
198     VOID defaultUnlockMutex(MUTEX mutex)
199     {

(gdb) f 3
#3  0x00043e9c in kinesisVideoStreamDefaultLockMutex (customData=18446744072314954400, mutex=18446744072315066176) at /<home>/webrtc-pusher/build/build.debug/amazonproducer/src/amazonproducer/dependency/libkvspic/kvspic-src/src/client/src/Callbacks.c:40
40          MUTEX_LOCK(mutex);
(gdb) list
35       * Default lock mutex functionality
36       */
37      VOID kinesisVideoStreamDefaultLockMutex(UINT64 customData, MUTEX mutex)
38      {
39          UNUSED_PARAM(customData);
40          MUTEX_LOCK(mutex);
41      }
42

(gdb) f 4
#4  0x00045678 in getKinesisVideoMetrics (clientHandle=18446744072315038536, pKinesisVideoMetrics=0xa31fb028) at /<home>/webrtc-pusher/build/build.debug/amazonproducer/src/amazonproducer/dependency/libkvspic/kvspic-src/src/client/src/Client.c:459
459         pKinesisVideoClient->clientCallbacks.lockMutexFn(pKinesisVideoClient->clientCallbacks.customData, pKinesisVideoClient->base.streamListLock);
(gdb) list
454         pKinesisVideoMetrics->totalTransferRate = 0;
455         pKinesisVideoMetrics->totalFrameRate = 0;
456         pKinesisVideoMetrics->totalElementaryFrameRate = 0;
457
458         // Lock the client streams list lock because we're iterating over
459         pKinesisVideoClient->clientCallbacks.lockMutexFn(pKinesisVideoClient->clientCallbacks.customData, pKinesisVideoClient->base.streamListLock);
460         locked = TRUE;
461
462         for (i = 0; i < pKinesisVideoClient->deviceInfo.streamCount; i++) {
463             if (NULL != pKinesisVideoClient->streams[i]) {

Run 2

Last log:

2022-02-18 21:31:02 DEBUG   postReadCallback(): Wrote 849 bytes to Kinesis Video. Upload stream handle: 0

[22:31:02] debug [   connection.c: 293] |> Start to read.
[22:31:02] debug [   connection.c: 299] <| Read 1451 bytes.
[22:31:02] debug [       pusher.c: 477] |> calling `putKinesisVideoFrame()` with 1421 bytes.
[22:31:02] debug [       pusher.c: 484] <| called `putKinesisVideoFrame()`.
[22:31:02] debug [   connection.c: 293] |> Start to read.
[22:31:02] debug [   connection.c: 299] <| Read 371 bytes.
[22:31:02] debug [       pusher.c: 477] |> calling `putKinesisVideoFrame()` with 341 bytes.
[22:31:02] debug [       pusher.c: 484] <| called `putKinesisVideoFrame()`.
[22:31:02] debug [   connection.c: 293] |> Start to read.
[22:31:02] debug [   connection.c: 299] <| Read 371 bytes.
[22:31:02] debug [       pusher.c: 477] |> calling `putKinesisVideoFrame()` with 341 bytes.
[22:31:02] debug [       pusher.c: 484] <| called `putKinesisVideoFrame()`.
2022-02-18 21:31:02 DEBUG   postReadCallback(): Wrote 2143 bytes to Kinesis Video. Upload stream handle: 0
[22:31:02] debug [   connection.c: 293] |> Start to read.[22:31:02] debug [   connection.c: 299] <| Read 84 bytes.
[22:31:02] debug [       pusher.c: 477] |> calling `putKinesisVideoFrame()` with 54 bytes.
[22:31:02] debug [       pusher.c: 484] <| called `putKinesisVideoFrame()`.
2022-02-18 21:31:02 DEBUG   postReadCallback(): Wrote 775 bytes to Kinesis Video. Upload stream handle: 0
[22:31:02] debug [   connection.c: 293] |> Start to read.
[22:31:02] debug [   connection.c: 299] <| Read 372 bytes.
[22:31:02] debug [       pusher.c: 477] |> calling `putKinesisVideoFrame()` with 342 bytes.
[22:31:02] debug [       pusher.c: 484] <| called `putKinesisVideoFrame()`.
[22:31:02] debug [   connection.c: 293] |> Start to read.
[22:31:02] debug [   connection.c: 299] <| Read 371 bytes.
[22:31:02] debug [       pusher.c: 477] |> calling `putKinesisVideoFrame()` with 341 bytes.
[22:31:02] debug [       pusher.c: 484] <| called `putKinesisVideoFrame()`.
[22:31:02] debug [   connection.c: 293] |> Start to read.
[22:31:02] debug [   connection.c: 299] <| Read 371 bytes.
[22:31:02] debug [       pusher.c: 477] |> calling `putKinesisVideoFrame()` with 341 bytes.
[22:31:02] debug [       pusher.c: 484] <| called `putKinesisVideoFrame()`.
[22:31:02] debug [   connection.c: 293] |> Start to read.
[22:31:02] debug [   connection.c: 299] <| Read 82 bytes.
[22:31:02] debug [       pusher.c: 477] |> calling `putKinesisVideoFrame()` with 52 bytes.
[22:31:02] debug [       pusher.c: 484] <| called `putKinesisVideoFrame()`.
2022-02-18 21:31:02 DEBUG   postReadCallback(): Wrote 774 bytes to Kinesis Video. Upload stream handle: 0
[22:31:02] debug [   connection.c: 293] |> Start to read.
[22:31:02] debug [   connection.c: 299] <| Read 1112 bytes.
[22:31:02] debug [       pusher.c: 477] |> calling `putKinesisVideoFrame()` with 1082 bytes.
[22:31:02] debug [       pusher.c: 484] <| called `putKinesisVideoFrame()`.
2022-02-18 21:31:02 DEBUG   postReadCallback(): Wrote 1449 bytes to Kinesis Video. Upload stream handle: 0
[22:31:02] debug [   connection.c: 293] |> Start to read.
[22:31:02] debug [   connection.c: 299] <| Read 372 bytes.
[22:31:02] debug [       pusher.c: 477] |> calling `putKinesisVideoFrame()` with 342 bytes.
[22:31:02] debug [       pusher.c: 484] <| called `putKinesisVideoFrame()`.
[22:31:02] debug [   connection.c: 293] |> Start to read.
[22:31:02] debug [   connection.c: 299] <| Read 371 bytes.
[22:31:02] debug [       pusher.c: 477] |> calling `putKinesisVideoFrame()` with 341 bytes.
[22:31:02] debug [       pusher.c: 484] <| called `putKinesisVideoFrame()`.
[22:31:02] debug [   connection.c: 293] |> Start to read.
[22:31:02] debug [   connection.c: 299] <| Read 371 bytes.
[22:31:02] debug [       pusher.c: 477] |> calling `putKinesisVideoFrame()` with 341 bytes.
[22:31:02] debug [       pusher.c: 484] <| called `putKinesisVideoFrame()`.
[22:31:02] debug [   connection.c: 293] |> Start to read.
[22:31:02] debug [   connection.c: 299] <| Read 26285 bytes.
[22:31:02] debug [       pusher.c: 477] |> calling `putKinesisVideoFrame()` with 26255 bytes.

2022-02-18 21:31:02 DEBUG   postReadCallback(): Wrote 709 bytes to Kinesis Video. Upload stream handle: 0
2022-02-18 21:31:03 DEBUG   postReadCallback(): Pausing CURL read for upload handle: 0
2022-02-18 21:31:38 WARN    curlCompleteSync(): curl perform failed for url https://s-0cb96ab8.kinesisvideo.eu-west-2.amazonaws.com/putMedia with result Timeout was reached: Operation too slow. Less than 30 bytes/sec transferred the last 30 seconds                            
                                                                       
2022-02-18 21:31:38 WARN    curlCompleteSync(): HTTP Error 0 : Response: (null)

Request URL: https://s-0cb96ab8.kinesisvideo.eu-west-2.amazonaws.com/putMedia
Request Headers:
    Authorization: AWS4-HMAC-SHA256 Credential=<cred>/20220218/eu-west-2/kinesisvideo/aws4_request, SignedHeaders=connection;host;transfer-encoding;user-agent;x-amz-date;x-amzn-fragment-acknowledgment-required;x-amzn-fragment-timecode-type;x-amzn-producer-start-timestamp;x-amzn-stream-name, Signature=<sig>
    connection: keep-alive
    content-type: application/json
    host: s-0cb96ab8.kinesisvideo.eu-west-2.amazonaws.com
    transfer-encoding: chunked
    user-agent: AWS-SDK-KVS/3.0.0 GCC/8.3.0 Linux/5.10.52-v7l+ armv7l
    X-Amz-Date: 20220218T213055Z
    x-amzn-fragment-acknowledgment-required: 1
    x-amzn-fragment-timecode-type: RELATIVE
    x-amzn-producer-start-timestamp: 1645219855.210
    x-amzn-stream-name: roxlu

2022-02-18 21:31:38 DEBUG   putStreamCurlHandler(): Network thread for Kinesis Video stream: roxlu with upload handle: 0 exited. http status: 0
2022-02-18 21:31:38 WARN    putStreamCurlHandler(): Stream with streamHandle 18446744072315065480 uploadHandle 0 has exited without triggering end-of-stream. Service call result: 599                                                                                                                                                                     
2022-02-18 21:31:38 INFO    kinesisVideoStreamTerminated(): Stream 0xfffffffface1d488 terminated upload handle 0 with service call result 599.
2022-02-18 21:31:38 DEBUG   defaultStreamStateTransitionHook(): Stream state machine retry count: 0
2022-02-18 21:31:38 DEBUG   defaultStreamStateTransitionHook():

 KinesisVideoStream base result is [599]. Executing KVS retry handler of retry strategy type [1]
[Thread 0xa27fea90 (LWP 18424) exited]

UPDATE As a test, I've disabled the getKinesisVideoMetrics() by returning STATUS_SUCCESS directly. My app is running for almost an hour w/o any freezes/hangs after this change. I'm still performing a long run test. Even if this might sovle the issue, I'm not entirely sure what happens; if this was a deadlock it would make sense, but from my GDB analysis this doesn't seem like a deadlock.

Activity icon
issue

roxlu issue comment awslabs/amazon-kinesis-video-streams-producer-c

roxlu
roxlu

[QUESTION] Thread safety: create `STREAM_HANDLE` in thread A and call `putKinesisVideoFrame()` from thread B.

I've been running into a situation where my application, that uses this KVS Producer SDK, suddenly hangs/freezes.

The interesting thing is that this only happens when I test on a RPI and not on my main intel i7 development PC. After inspecting the state with GDB I noticed something which looks like two threads wait on the same lock, but I'm not yet 100% sure. I'm creating a STREAM_HANDLE instance in thread A, then I start a thread which collects RTP data and then calls putKinesisVideoFrame(). Once the stream has started the STREAM_HANDLE is not used by any other thread from my app. As there are no data-races, at least not that I'm aware off, this shouldn't be a problem. But to be sure I synchronise access to my STREAM_HANDLE using a mutex.

I've collected some data and I'm wondering if my app "freezes" on these locks. When you look below at Thread 4, KVS-1 and the call to lockMutexFn(pKinesisVideoClient->clientCallbacks.customData, ...) and Thread 9, KVS-2 you can see that they both try to lock the same mutex.

--

Run 1

The last output looks like:

2022-02-18 20:48:02 DEBUG   postReadCallback(): Wrote 843 bytes to Kinesis Video. Upload stream handle: 0                                                                 

[21:48:02] debug [   connection.c: 293] |> Start to read.
[21:48:02] debug [   connection.c: 299] <| Read 26873 bytes.
[21:48:02] debug [       pusher.c: 464] |> calling `putKinesisVideoFrame()` with 26843 bytes.                                                                             

2022-02-18 20:48:02 DEBUG   postReadCallback(): Wrote 708 bytes to Kinesis Video. Upload stream handle: 0                                                                 
2022-02-18 20:48:02 DEBUG   postReadCallback(): Pausing CURL read for upload handle: 0                                                                                    
2022-02-18 20:48:37 WARN    curlCompleteSync(): curl perform failed for url https://s-0cb96ab8.kinesisvideo.eu-west-2.amazonaws.com/putMedia with result Timeout was reached: Operation too slow. Less than 30 bytes/sec transferred the last 30 seconds
2022-02-18 20:48:37 WARN    curlCompleteSync(): HTTP Error 0 : Response: (null)

Request URL: https://s-0cb96ab8.kinesisvideo.eu-west-2.amazonaws.com/putMedia
Request Headers:
    Authorization: AWS4-HMAC-SHA256 Credential=<cred>/20220218/eu-west-2/kinesisvideo/aws4_request, SignedHeaders=connection;host;transfer-encoding;user-agent;x-amz-date;x-amzn-fragment-acknowledgment-required;x-amzn-fragment-timecode-type;x-amzn-producer-start-timestamp;x-amzn-stream-name, Signature=<sig>
    connection: keep-alive
    content-type: application/json
    host: s-0cb96ab8.kinesisvideo.eu-west-2.amazonaws.com
    transfer-encoding: chunked
    user-agent: AWS-SDK-KVS/3.0.0 GCC/8.3.0 Linux/5.10.52-v7l+ armv7l
    X-Amz-Date: 20220218T204024Z
    x-amzn-fragment-acknowledgment-required: 1
    x-amzn-fragment-timecode-type: RELATIVE
    x-amzn-producer-start-timestamp: 1645216824.732
    x-amzn-stream-name: roxlu
    
2022-02-18 20:48:37 DEBUG   putStreamCurlHandler(): Network thread for Kinesis Video stream: roxlu with upload handle: 0 exited. http status: 0
2022-02-18 20:48:37 WARN    putStreamCurlHandler(): Stream with streamHandle 18446744072315065456 uploadHandle 0 has exited without triggering end-of-stream. Service call result: 599
2022-02-18 20:48:37 INFO    kinesisVideoStreamTerminated(): Stream 0xfffffffface1d470 terminated upload handle 0 with service call result 599.
2022-02-18 20:48:37 DEBUG   defaultStreamStateTransitionHook(): Stream state machine retry count: 0
2022-02-18 20:48:37 DEBUG   defaultStreamStateTransitionHook():
 
 KinesisVideoStream base result is [599]. Executing KVS retry handler of retry strategy type [1]
 
[Thread 0xa27fea90 (LWP 17414) exited]

Using GDB I can see there are several threads:

(gdb) info threads 
  Id   Target Id                                    Frame 
  1    Thread 0xae1e6920 (LWP 17402) "pusher-debug" 0xb5574a3c in __GI___pthread_timedjoin_ex (threadid=2910841488, [email protected]=0xbeffeaa8, [email protected]=0x0, [email protected]=true) at pthread_join_common.c:89
  2    Thread 0xae1a1a90 (LWP 17406) "pusher-debug" futex_wait_cancelable (private=0, expected=0, futex_word=0x3250b0 <glog+104>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  3    Thread 0xad7fea90 (LWP 17407) "pusher-debug" __GI___select (timeout=0xbeffea4a, exceptfds=0x0, writefds=0x0, readfds=0xad7fe3ac, nfds=5) at ../sysdeps/unix/sysv/linux/select.c:41
  4    Thread 0xa39fca90 (LWP 17408) "pusher-debug" __lll_lock_wait ([email protected]=0xace24980, private=0) at lowlevellock.c:46
* 9    Thread 0xa31fba90 (LWP 17413) "pusher-debug" __lll_lock_wait ([email protected]=0xace1d740, private=0) at lowlevellock.c:46
  • Thread 1: this main, which joins my server/application thread.
  • Thread 2: this is my logger thread; waits for a condition to signal
  • Thread 3: the server thread; accepts a connection on which is receives media packets

Thread 1, main:

(gdb) thread 1
[Switching to thread 1 (Thread 0xae1e6920 (LWP 17402))]
#0  0xb5574a3c in __GI___pthread_timedjoin_ex (threadid=2910841488, [email protected]=0xbeffeaa8, [email protected]=0x0, [email protected]=true) at pthread_join_common.c:89
89      pthread_join_common.c: No such file or directory.

(gdb) bt
#0  0xb5574a3c in __GI___pthread_timedjoin_ex (threadid=2910841488, [email protected]=0xbeffeaa8, [email protected]=0x0, [email protected]=true) at pthread_join_common.c:89
#1  0xb5581008 in thrd_join (thr=<optimized out>, res=0x0) at thrd_join.c:25
#2  0x0001a2c8 in rx_pusher_server_join (ctx=0x354130) at /<home>/webrtc-pusher/src/rx/pusher/server.c:187
#3  0x00015550 in main (argc=5, argv=0xbeffec74) at /<home>/webrtc-pusher/src/app/pusher.c:83

Thread 2, logger:

#0  0xb557a1a0 in futex_wait_cancelable (private=0, expected=0, futex_word=0x3250b0 <glog+104>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  0xb557a1a0 in __pthread_cond_wait_common (abstime=0x0, mutex=0x0, cond=0x325088 <glog+64>) at pthread_cond_wait.c:502
#2  0xb557a1a0 in __pthread_cond_wait (cond=0x325088 <glog+64>, mutex=0x0) at pthread_cond_wait.c:655
#3  0xb5581388 in cnd_wait (cond=<optimized out>, mutex=<optimized out>) at cnd_wait.c:24
#4  0x0001697c in rx_log_thread (user=0x325048 <glog>) at /<home>/src/log.c:665
#5  0xb5573494 in start_thread (arg=0xae1a1a90) at pthread_create.c:486
#6  0xb54f6578 in  () at ../sysdeps/unix/sysv/linux/arm/clone.S:73

Thread 3, server thread that accepts client connections:

(gdb) thread 3

[Switching to thread 3 (Thread 0xad7fea90 (LWP 17407))]
#0  __GI___select (timeout=0xbeffea4a, exceptfds=0x0, writefds=0x0, readfds=0xad7fe3ac, nfds=5) at ../sysdeps/unix/sysv/linux/select.c:41
41      ../sysdeps/unix/sysv/linux/select.c: No such file or directory.

(gdb) list

36      in ../sysdeps/unix/sysv/linux/select.c

(gdb) bt

#0  0xb54ef6ac in __GI___select (timeout=0xbeffea4a, exceptfds=0x0, writefds=0x0, readfds=0xad7fe3ac, nfds=5) at ../sysdeps/unix/sysv/linux/select.c:41
#1  0xb54ef6ac in __GI___select (nfds=5, readfds=0xad7fe3ac, writefds=0x0, exceptfds=0x0, timeout=0xad7fe42c) at ../sysdeps/unix/sysv/linux/select.c:37
#2  0x0001d4f8 in rx_socket_can_read (ctx=0x33c0e8, timeoutSec=3, timeoutMicroSec=500000, result=0xad7fe467 "\003\310\005ସ", <incomplete sequence \340\254>) at /<home>/webrtc-pusher/src/rx/socket.c:692
#3  0x0001a758 in server_thread_func (user=0x354130) at /<home>/webrtc-pusher/src/rx/pusher/server.c:317
#4  0xb5573494 in start_thread (arg=0xad7fea90) at pthread_create.c:486
#5  0xb54f6578 in  () at ../sysdeps/unix/sysv/linux/arm/clone.S:73
(gdb) f 2
#2  0x0001d4f8 in rx_socket_can_read (ctx=0x33c0e8, timeoutSec=3, timeoutMicroSec=500000, result=0xad7fe467 "\003\310\005ସ", <incomplete sequence \340\254>) at /<home>/webrtc-pusher/src/rx/socket.c:692
692       r = select(ctx->handle + 1, &readset, NULL, NULL, &timeout);

(gdb) list
687       timeout.tv_usec = timeoutMicroSec;
688       
689       FD_ZERO(&readset);
690       FD_SET(ctx->handle, &readset);
691
692       r = select(ctx->handle + 1, &readset, NULL, NULL, &timeout);
693       
694       if (0 == r) {
695         /* Nope, timed out. */
696         *result = RX_SOCK_TIMEOUT;

Thread 4, KVS-1

(gdb) thread 4
[Switching to thread 4 (Thread 0xa39fca90 (LWP 17408))]
#0  __lll_lock_wait ([email protected]=0xace24980, private=0) at lowlevellock.c:46
46      lowlevellock.c: No such file or directory.

(gdb) bt
#0  0xb557de98 in __lll_lock_wait ([email protected]=0xace24980, private=0) at lowlevellock.c:46
#1  0xb5575f58 in __GI___pthread_mutex_lock (mutex=0xace24980) at pthread_mutex_lock.c:115
#2  0x00063318 in defaultLockMutex (mutex=18446744072315095424) at /<home>/webrtc-pusher/build/build.debug/amazonproducer/src/amazonproducer/dependency/libkvspic/kvspic-src/src/utils/src/Mutex.c:195
#3  0x00043e9c in kinesisVideoStreamDefaultLockMutex (customData=18446744072314954400, mutex=18446744072315095424) at /<home>/webrtc-pusher/build/build.debug/amazonproducer/src/amazonproducer/dependency/libkvspic/kvspic-src/src/client/src/Callbacks.c:40
#4  0x000443c0 in checkIntermittentProducerCallback (timerId=0, currentTime=16452172820799116, customData=18446744072315038536) at /<home>/webrtc-pusher/build/build.debug/amazonproducer/src/amazonproducer/dependency/libkvspic/kvspic-src/src/client/src/Client.c:64
#5  0x00069a8c in timerQueueExecutor (args=0xace1d8b0) at /<home>/webrtc-pusher/build/build.debug/amazonproducer/src/amazonproducer/dependency/libkvspic/kvspic-src/src/utils/src/TimerQueue.c:560
#6  0xb5573494 in start_thread (arg=0xa39fca90) at pthread_create.c:486
#7  0xb54f6578 in  () at ../sysdeps/unix/sysv/linux/arm/clone.S:73

(gdb) f 3
#3  0x00043e9c in kinesisVideoStreamDefaultLockMutex (customData=18446744072314954400, mutex=18446744072315095424) at /<home>/webrtc-pusher/build/build.debug/amazonproducer/src/amazonproducer/dependency/libkvspic/kvspic-src/src/client/src/Callbacks.c:40
40          MUTEX_LOCK(mutex);
(gdb) list
35       * Default lock mutex functionality
36       */
37      VOID kinesisVideoStreamDefaultLockMutex(UINT64 customData, MUTEX mutex)
38      {
39          UNUSED_PARAM(customData);
40          MUTEX_LOCK(mutex);
41      }
42
43      /**

(gdb) f 4
#4  0x000443c0 in checkIntermittentProducerCallback (timerId=0, currentTime=16452172820799116, customData=18446744072315038536) at /<home>/webrtc-pusher/build/build.debug/amazonproducer/src/amazonproducer/dependency/libkvspic/kvspic-src/src/client/src/Client.c:64
64                          pKinesisVideoClient->clientCallbacks.lockMutexFn(pKinesisVideoClient->clientCallbacks.customData, pFrameOrderCoordinator->lock);
(gdb) list
59                      pCurrStream = pKinesisVideoClient->streams[i];
60                      pFrameOrderCoordinator = pCurrStream->pFrameOrderCoordinator;
61                      if (pCurrStream->streamInfo.streamCaps.frameOrderingMode != FRAME_ORDER_MODE_PASS_THROUGH) {
62                          // In the case that frameOrderingMode = FRAME_ORDER_MODE_PASS_THROUGH, pFrameOrderCoordinator is NULL
63                          // so we will segfault, so the check for the frameOrderingMode is important
64                          pKinesisVideoClient->clientCallbacks.lockMutexFn(pKinesisVideoClient->clientCallbacks.customData, pFrameOrderCoordinator->lock);
65                          frameOrderCoordinatorLocked = TRUE;
66                      }
67                      // Lock the Stream
68                      pKinesisVideoClient->clientCallbacks.lockMutexFn(pKinesisVideoClient->clientCallbacks.customData, pCurrStream->base.lock);

Thread 9, KVS-2

(gdb) thread 9
[Switching to thread 9 (Thread 0xa31fba90 (LWP 17413))]
#0  __lll_lock_wait ([email protected]=0xace1d740, private=0) at lowlevellock.c:46
46      lowlevellock.c: No such file or directory.

(gdb) bt
#0  0xb557de98 in __lll_lock_wait ([email protected]=0xace1d740, private=0) at lowlevellock.c:46
#1  0xb5575f58 in __GI___pthread_mutex_lock (mutex=0xace1d740) at pthread_mutex_lock.c:115
#2  0x00063318 in defaultLockMutex (mutex=18446744072315066176) at /<home>/webrtc-pusher/build/build.debug/amazonproducer/src/amazonproducer/dependency/libkvspic/kvspic-src/src/utils/src/Mutex.c:195
#3  0x00043e9c in kinesisVideoStreamDefaultLockMutex (customData=18446744072314954400, mutex=18446744072315066176) at /<home>/webrtc-pusher/build/build.debug/amazonproducer/src/amazonproducer/dependency/libkvspic/kvspic-src/src/client/src/Callbacks.c:40
#4  0x00045678 in getKinesisVideoMetrics (clientHandle=18446744072315038536, pKinesisVideoMetrics=0xa31fb028) at /<home>/webrtc-pusher/build/build.debug/amazonproducer/src/amazonproducer/dependency/libkvspic/kvspic-src/src/client/src/Client.c:459
#5  0x0004f56c in logStreamMetric (pKinesisVideoStream=0xace1e018) at /<home>/webrtc-pusher/build/build.debug/amazonproducer/src/amazonproducer/dependency/libkvspic/kvspic-src/src/client/src/Stream.c:674
#6  0x00050e50 in putFrame (pKinesisVideoStream=0xace1e018, pFrame=0xa1e1b488) at /<home>/webrtc-pusher/build/build.debug/amazonproducer/src/amazonproducer/dependency/libkvspic/kvspic-src/src/client/src/Stream.c:991
#7  0x0004c048 in putEarliestFrame (pKinesisVideoStream=0xace1e018) at /<home>/webrtc-pusher/build/build.debug/amazonproducer/src/amazonproducer/dependency/libkvspic/kvspic-src/src/client/src/FrameOrderCoordinator.c:233
#8  0x0004c4bc in frameOrderCoordinatorPutFrame (pKinesisVideoStream=0xace1e018, pUserFrame=0xa31fb3f8) at /<home>/webrtc-pusher/build/build.debug/amazonproducer/src/amazonproducer/dependency/libkvspic/kvspic-src/src/client/src/FrameOrderCoordinator.c:319
#9  0x00046d14 in putKinesisVideoFrame (streamHandle=18446744072315065456, pFrame=0xa31fb3f8) at /<home>/webrtc-pusher/build/build.debug/amazonproducer/src/amazonproducer/dependency/libkvspic/kvspic-src/src/client/src/Client.c:876
#10 0x00022478 in rx_pusher_send_frame (ctx=0xace00610, frame=0xa31fb3f8) at /<home>/webrtc-pusher/src/rx/pusher.c:465
#11 0x0001b9bc in connection_thread_func (user=0xace005c8) at /<home>/webrtc-pusher/src/rx/pusher/connection.c:354
#12 0xb5573494 in start_thread (arg=0xa31fba90) at pthread_create.c:486
#13 0xb54f6578 in  () at ../sysdeps/unix/sysv/linux/arm/clone.S:73

(gdb) f 2
#2  0x00063318 in defaultLockMutex (mutex=18446744072315066176) at /<home>/webrtc-pusher/build/build.debug/amazonproducer/src/amazonproducer/dependency/libkvspic/kvspic-src/src/utils/src/Mutex.c:195
195         pthread_mutex_lock((pthread_mutex_t*) mutex);
(gdb) list
190         return (MUTEX) pMutex;
191     }
192
193     VOID defaultLockMutex(MUTEX mutex)
194     {
195         pthread_mutex_lock((pthread_mutex_t*) mutex);
196     }
197
198     VOID defaultUnlockMutex(MUTEX mutex)
199     {

(gdb) f 3
#3  0x00043e9c in kinesisVideoStreamDefaultLockMutex (customData=18446744072314954400, mutex=18446744072315066176) at /<home>/webrtc-pusher/build/build.debug/amazonproducer/src/amazonproducer/dependency/libkvspic/kvspic-src/src/client/src/Callbacks.c:40
40          MUTEX_LOCK(mutex);
(gdb) list
35       * Default lock mutex functionality
36       */
37      VOID kinesisVideoStreamDefaultLockMutex(UINT64 customData, MUTEX mutex)
38      {
39          UNUSED_PARAM(customData);
40          MUTEX_LOCK(mutex);
41      }
42

(gdb) f 4
#4  0x00045678 in getKinesisVideoMetrics (clientHandle=18446744072315038536, pKinesisVideoMetrics=0xa31fb028) at /<home>/webrtc-pusher/build/build.debug/amazonproducer/src/amazonproducer/dependency/libkvspic/kvspic-src/src/client/src/Client.c:459
459         pKinesisVideoClient->clientCallbacks.lockMutexFn(pKinesisVideoClient->clientCallbacks.customData, pKinesisVideoClient->base.streamListLock);
(gdb) list
454         pKinesisVideoMetrics->totalTransferRate = 0;
455         pKinesisVideoMetrics->totalFrameRate = 0;
456         pKinesisVideoMetrics->totalElementaryFrameRate = 0;
457
458         // Lock the client streams list lock because we're iterating over
459         pKinesisVideoClient->clientCallbacks.lockMutexFn(pKinesisVideoClient->clientCallbacks.customData, pKinesisVideoClient->base.streamListLock);
460         locked = TRUE;
461
462         for (i = 0; i < pKinesisVideoClient->deviceInfo.streamCount; i++) {
463             if (NULL != pKinesisVideoClient->streams[i]) {

Run 2

Last log:

2022-02-18 21:31:02 DEBUG   postReadCallback(): Wrote 849 bytes to Kinesis Video. Upload stream handle: 0

[22:31:02] debug [   connection.c: 293] |> Start to read.
[22:31:02] debug [   connection.c: 299] <| Read 1451 bytes.
[22:31:02] debug [       pusher.c: 477] |> calling `putKinesisVideoFrame()` with 1421 bytes.
[22:31:02] debug [       pusher.c: 484] <| called `putKinesisVideoFrame()`.
[22:31:02] debug [   connection.c: 293] |> Start to read.
[22:31:02] debug [   connection.c: 299] <| Read 371 bytes.
[22:31:02] debug [       pusher.c: 477] |> calling `putKinesisVideoFrame()` with 341 bytes.
[22:31:02] debug [       pusher.c: 484] <| called `putKinesisVideoFrame()`.
[22:31:02] debug [   connection.c: 293] |> Start to read.
[22:31:02] debug [   connection.c: 299] <| Read 371 bytes.
[22:31:02] debug [       pusher.c: 477] |> calling `putKinesisVideoFrame()` with 341 bytes.
[22:31:02] debug [       pusher.c: 484] <| called `putKinesisVideoFrame()`.
2022-02-18 21:31:02 DEBUG   postReadCallback(): Wrote 2143 bytes to Kinesis Video. Upload stream handle: 0
[22:31:02] debug [   connection.c: 293] |> Start to read.[22:31:02] debug [   connection.c: 299] <| Read 84 bytes.
[22:31:02] debug [       pusher.c: 477] |> calling `putKinesisVideoFrame()` with 54 bytes.
[22:31:02] debug [       pusher.c: 484] <| called `putKinesisVideoFrame()`.
2022-02-18 21:31:02 DEBUG   postReadCallback(): Wrote 775 bytes to Kinesis Video. Upload stream handle: 0
[22:31:02] debug [   connection.c: 293] |> Start to read.
[22:31:02] debug [   connection.c: 299] <| Read 372 bytes.
[22:31:02] debug [       pusher.c: 477] |> calling `putKinesisVideoFrame()` with 342 bytes.
[22:31:02] debug [       pusher.c: 484] <| called `putKinesisVideoFrame()`.
[22:31:02] debug [   connection.c: 293] |> Start to read.
[22:31:02] debug [   connection.c: 299] <| Read 371 bytes.
[22:31:02] debug [       pusher.c: 477] |> calling `putKinesisVideoFrame()` with 341 bytes.
[22:31:02] debug [       pusher.c: 484] <| called `putKinesisVideoFrame()`.
[22:31:02] debug [   connection.c: 293] |> Start to read.
[22:31:02] debug [   connection.c: 299] <| Read 371 bytes.
[22:31:02] debug [       pusher.c: 477] |> calling `putKinesisVideoFrame()` with 341 bytes.
[22:31:02] debug [       pusher.c: 484] <| called `putKinesisVideoFrame()`.
[22:31:02] debug [   connection.c: 293] |> Start to read.
[22:31:02] debug [   connection.c: 299] <| Read 82 bytes.
[22:31:02] debug [       pusher.c: 477] |> calling `putKinesisVideoFrame()` with 52 bytes.
[22:31:02] debug [       pusher.c: 484] <| called `putKinesisVideoFrame()`.
2022-02-18 21:31:02 DEBUG   postReadCallback(): Wrote 774 bytes to Kinesis Video. Upload stream handle: 0
[22:31:02] debug [   connection.c: 293] |> Start to read.
[22:31:02] debug [   connection.c: 299] <| Read 1112 bytes.
[22:31:02] debug [       pusher.c: 477] |> calling `putKinesisVideoFrame()` with 1082 bytes.
[22:31:02] debug [       pusher.c: 484] <| called `putKinesisVideoFrame()`.
2022-02-18 21:31:02 DEBUG   postReadCallback(): Wrote 1449 bytes to Kinesis Video. Upload stream handle: 0
[22:31:02] debug [   connection.c: 293] |> Start to read.
[22:31:02] debug [   connection.c: 299] <| Read 372 bytes.
[22:31:02] debug [       pusher.c: 477] |> calling `putKinesisVideoFrame()` with 342 bytes.
[22:31:02] debug [       pusher.c: 484] <| called `putKinesisVideoFrame()`.
[22:31:02] debug [   connection.c: 293] |> Start to read.
[22:31:02] debug [   connection.c: 299] <| Read 371 bytes.
[22:31:02] debug [       pusher.c: 477] |> calling `putKinesisVideoFrame()` with 341 bytes.
[22:31:02] debug [       pusher.c: 484] <| called `putKinesisVideoFrame()`.
[22:31:02] debug [   connection.c: 293] |> Start to read.
[22:31:02] debug [   connection.c: 299] <| Read 371 bytes.
[22:31:02] debug [       pusher.c: 477] |> calling `putKinesisVideoFrame()` with 341 bytes.
[22:31:02] debug [       pusher.c: 484] <| called `putKinesisVideoFrame()`.
[22:31:02] debug [   connection.c: 293] |> Start to read.
[22:31:02] debug [   connection.c: 299] <| Read 26285 bytes.
[22:31:02] debug [       pusher.c: 477] |> calling `putKinesisVideoFrame()` with 26255 bytes.

2022-02-18 21:31:02 DEBUG   postReadCallback(): Wrote 709 bytes to Kinesis Video. Upload stream handle: 0
2022-02-18 21:31:03 DEBUG   postReadCallback(): Pausing CURL read for upload handle: 0
2022-02-18 21:31:38 WARN    curlCompleteSync(): curl perform failed for url https://s-0cb96ab8.kinesisvideo.eu-west-2.amazonaws.com/putMedia with result Timeout was reached: Operation too slow. Less than 30 bytes/sec transferred the last 30 seconds                            
                                                                       
2022-02-18 21:31:38 WARN    curlCompleteSync(): HTTP Error 0 : Response: (null)

Request URL: https://s-0cb96ab8.kinesisvideo.eu-west-2.amazonaws.com/putMedia
Request Headers:
    Authorization: AWS4-HMAC-SHA256 Credential=<cred>/20220218/eu-west-2/kinesisvideo/aws4_request, SignedHeaders=connection;host;transfer-encoding;user-agent;x-amz-date;x-amzn-fragment-acknowledgment-required;x-amzn-fragment-timecode-type;x-amzn-producer-start-timestamp;x-amzn-stream-name, Signature=<sig>
    connection: keep-alive
    content-type: application/json
    host: s-0cb96ab8.kinesisvideo.eu-west-2.amazonaws.com
    transfer-encoding: chunked
    user-agent: AWS-SDK-KVS/3.0.0 GCC/8.3.0 Linux/5.10.52-v7l+ armv7l
    X-Amz-Date: 20220218T213055Z
    x-amzn-fragment-acknowledgment-required: 1
    x-amzn-fragment-timecode-type: RELATIVE
    x-amzn-producer-start-timestamp: 1645219855.210
    x-amzn-stream-name: roxlu

2022-02-18 21:31:38 DEBUG   putStreamCurlHandler(): Network thread for Kinesis Video stream: roxlu with upload handle: 0 exited. http status: 0
2022-02-18 21:31:38 WARN    putStreamCurlHandler(): Stream with streamHandle 18446744072315065480 uploadHandle 0 has exited without triggering end-of-stream. Service call result: 599                                                                                                                                                                     
2022-02-18 21:31:38 INFO    kinesisVideoStreamTerminated(): Stream 0xfffffffface1d488 terminated upload handle 0 with service call result 599.
2022-02-18 21:31:38 DEBUG   defaultStreamStateTransitionHook(): Stream state machine retry count: 0
2022-02-18 21:31:38 DEBUG   defaultStreamStateTransitionHook():

 KinesisVideoStream base result is [599]. Executing KVS retry handler of retry strategy type [1]
[Thread 0xa27fea90 (LWP 18424) exited]

UPDATE As a test, I've disabled the getKinesisVideoMetrics() by returning STATUS_SUCCESS directly. My app is running for almost an hour w/o any freezes/hangs after this change. I'm still performing a long run test. Even if this might sovle the issue, I'm not entirely sure what happens; if this was a deadlock it would make sense, but from my GDB analysis this doesn't seem like a deadlock.

roxlu
roxlu

@MushMal, I've solved this by returning STATUS_SUCCESS from getKinesisVideoMetrics() w/o directly executing the rest of the function and since then (1000s of streams, ~15days) I haven't encountered this issue again. For now this is OK for me and I'll close this issue.

Mar
8
2 months ago
Activity icon
issue

roxlu issue awslabs/amazon-kinesis-video-streams-webrtc-sdk-c

roxlu
roxlu

[QUESTION] How to deal with symmetric NATs for viewers?

I've got an WebRtc app where viewers watch a camera stream. The viewers only ingest media from the server and don't send any media. We noticed that when a viewer uses 4G they are often behind a symmetric NAT which triggers the use of a TURN server. The server uses a port/address restricted cone NAT that we can configure.

I'm pretty sure that we need TURN, but I'm wondering if there might be a solution to setup a peer-2-peer connection. My understanding is that the viewer who uses e.g. Chrome binds an UDP socket on a particular port and uses recvfrom() to read the media packets from the server that's sending the packets. Because no NAT entry can be used from the symmetric NAT of the viewer, we need an intermediate TURN/relay.

Feb
23
2 months ago
started
started time in 2 months ago
Feb
22
2 months ago
Activity icon
issue

roxlu issue comment awslabs/amazon-kinesis-video-streams-producer-c

roxlu
roxlu

[QUESTION] Thread safety: create `STREAM_HANDLE` in thread A and call `putKinesisVideoFrame()` from thread B.

I've been running into a situation where my application, that uses this KVS Producer SDK, suddenly hangs/freezes.

The interesting thing is that this only happens when I test on a RPI and not on my main intel i7 development PC. After inspecting the state with GDB I noticed something which looks like two threads wait on the same lock, but I'm not yet 100% sure. I'm creating a STREAM_HANDLE instance in thread A, then I start a thread which collects RTP data and then calls putKinesisVideoFrame(). Once the stream has started the STREAM_HANDLE is not used by any other thread from my app. As there are no data-races, at least not that I'm aware off, this shouldn't be a problem. But to be sure I synchronise access to my STREAM_HANDLE using a mutex.

I've collected some data and I'm wondering if my app "freezes" on these locks. When you look below at Thread 4, KVS-1 and the call to lockMutexFn(pKinesisVideoClient->clientCallbacks.customData, ...) and Thread 9, KVS-2 you can see that they both try to lock the same mutex.

--

Run 1

The last output looks like:

2022-02-18 20:48:02 DEBUG   postReadCallback(): Wrote 843 bytes to Kinesis Video. Upload stream handle: 0                                                                 

[21:48:02] debug [   connection.c: 293] |> Start to read.
[21:48:02] debug [   connection.c: 299] <| Read 26873 bytes.
[21:48:02] debug [       pusher.c: 464] |> calling `putKinesisVideoFrame()` with 26843 bytes.                                                                             

2022-02-18 20:48:02 DEBUG   postReadCallback(): Wrote 708 bytes to Kinesis Video. Upload stream handle: 0                                                                 
2022-02-18 20:48:02 DEBUG   postReadCallback(): Pausing CURL read for upload handle: 0                                                                                    
2022-02-18 20:48:37 WARN    curlCompleteSync(): curl perform failed for url https://s-0cb96ab8.kinesisvideo.eu-west-2.amazonaws.com/putMedia with result Timeout was reached: Operation too slow. Less than 30 bytes/sec transferred the last 30 seconds
2022-02-18 20:48:37 WARN    curlCompleteSync(): HTTP Error 0 : Response: (null)

Request URL: https://s-0cb96ab8.kinesisvideo.eu-west-2.amazonaws.com/putMedia
Request Headers:
    Authorization: AWS4-HMAC-SHA256 Credential=<cred>/20220218/eu-west-2/kinesisvideo/aws4_request, SignedHeaders=connection;host;transfer-encoding;user-agent;x-amz-date;x-amzn-fragment-acknowledgment-required;x-amzn-fragment-timecode-type;x-amzn-producer-start-timestamp;x-amzn-stream-name, Signature=<sig>
    connection: keep-alive
    content-type: application/json
    host: s-0cb96ab8.kinesisvideo.eu-west-2.amazonaws.com
    transfer-encoding: chunked
    user-agent: AWS-SDK-KVS/3.0.0 GCC/8.3.0 Linux/5.10.52-v7l+ armv7l
    X-Amz-Date: 20220218T204024Z
    x-amzn-fragment-acknowledgment-required: 1
    x-amzn-fragment-timecode-type: RELATIVE
    x-amzn-producer-start-timestamp: 1645216824.732
    x-amzn-stream-name: roxlu
    
2022-02-18 20:48:37 DEBUG   putStreamCurlHandler(): Network thread for Kinesis Video stream: roxlu with upload handle: 0 exited. http status: 0
2022-02-18 20:48:37 WARN    putStreamCurlHandler(): Stream with streamHandle 18446744072315065456 uploadHandle 0 has exited without triggering end-of-stream. Service call result: 599
2022-02-18 20:48:37 INFO    kinesisVideoStreamTerminated(): Stream 0xfffffffface1d470 terminated upload handle 0 with service call result 599.
2022-02-18 20:48:37 DEBUG   defaultStreamStateTransitionHook(): Stream state machine retry count: 0
2022-02-18 20:48:37 DEBUG   defaultStreamStateTransitionHook():
 
 KinesisVideoStream base result is [599]. Executing KVS retry handler of retry strategy type [1]
 
[Thread 0xa27fea90 (LWP 17414) exited]

Using GDB I can see there are several threads:

(gdb) info threads 
  Id   Target Id                                    Frame 
  1    Thread 0xae1e6920 (LWP 17402) "pusher-debug" 0xb5574a3c in __GI___pthread_timedjoin_ex (threadid=2910841488, [email protected]=0xbeffeaa8, [email protected]=0x0, [email protected]=true) at pthread_join_common.c:89
  2    Thread 0xae1a1a90 (LWP 17406) "pusher-debug" futex_wait_cancelable (private=0, expected=0, futex_word=0x3250b0 <glog+104>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  3    Thread 0xad7fea90 (LWP 17407) "pusher-debug" __GI___select (timeout=0xbeffea4a, exceptfds=0x0, writefds=0x0, readfds=0xad7fe3ac, nfds=5) at ../sysdeps/unix/sysv/linux/select.c:41
  4    Thread 0xa39fca90 (LWP 17408) "pusher-debug" __lll_lock_wait ([email protected]=0xace24980, private=0) at lowlevellock.c:46
* 9    Thread 0xa31fba90 (LWP 17413) "pusher-debug" __lll_lock_wait ([email protected]=0xace1d740, private=0) at lowlevellock.c:46
  • Thread 1: this main, which joins my server/application thread.
  • Thread 2: this is my logger thread; waits for a condition to signal
  • Thread 3: the server thread; accepts a connection on which is receives media packets

Thread 1, main:

(gdb) thread 1
[Switching to thread 1 (Thread 0xae1e6920 (LWP 17402))]
#0  0xb5574a3c in __GI___pthread_timedjoin_ex (threadid=2910841488, [email protected]=0xbeffeaa8, [email protected]=0x0, [email protected]=true) at pthread_join_common.c:89
89      pthread_join_common.c: No such file or directory.

(gdb) bt
#0  0xb5574a3c in __GI___pthread_timedjoin_ex (threadid=2910841488, [email protected]=0xbeffeaa8, [email protected]=0x0, [email protected]=true) at pthread_join_common.c:89
#1  0xb5581008 in thrd_join (thr=<optimized out>, res=0x0) at thrd_join.c:25
#2  0x0001a2c8 in rx_pusher_server_join (ctx=0x354130) at /<home>/webrtc-pusher/src/rx/pusher/server.c:187
#3  0x00015550 in main (argc=5, argv=0xbeffec74) at /<home>/webrtc-pusher/src/app/pusher.c:83

Thread 2, logger:

#0  0xb557a1a0 in futex_wait_cancelable (private=0, expected=0, futex_word=0x3250b0 <glog+104>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  0xb557a1a0 in __pthread_cond_wait_common (abstime=0x0, mutex=0x0, cond=0x325088 <glog+64>) at pthread_cond_wait.c:502
#2  0xb557a1a0 in __pthread_cond_wait (cond=0x325088 <glog+64>, mutex=0x0) at pthread_cond_wait.c:655
#3  0xb5581388 in cnd_wait (cond=<optimized out>, mutex=<optimized out>) at cnd_wait.c:24
#4  0x0001697c in rx_log_thread (user=0x325048 <glog>) at /<home>/src/log.c:665
#5  0xb5573494 in start_thread (arg=0xae1a1a90) at pthread_create.c:486
#6  0xb54f6578 in  () at ../sysdeps/unix/sysv/linux/arm/clone.S:73

Thread 3, server thread that accepts client connections:

(gdb) thread 3

[Switching to thread 3 (Thread 0xad7fea90 (LWP 17407))]
#0  __GI___select (timeout=0xbeffea4a, exceptfds=0x0, writefds=0x0, readfds=0xad7fe3ac, nfds=5) at ../sysdeps/unix/sysv/linux/select.c:41
41      ../sysdeps/unix/sysv/linux/select.c: No such file or directory.

(gdb) list

36      in ../sysdeps/unix/sysv/linux/select.c

(gdb) bt

#0  0xb54ef6ac in __GI___select (timeout=0xbeffea4a, exceptfds=0x0, writefds=0x0, readfds=0xad7fe3ac, nfds=5) at ../sysdeps/unix/sysv/linux/select.c:41
#1  0xb54ef6ac in __GI___select (nfds=5, readfds=0xad7fe3ac, writefds=0x0, exceptfds=0x0, timeout=0xad7fe42c) at ../sysdeps/unix/sysv/linux/select.c:37
#2  0x0001d4f8 in rx_socket_can_read (ctx=0x33c0e8, timeoutSec=3, timeoutMicroSec=500000, result=0xad7fe467 "\003\310\005ସ", <incomplete sequence \340\254>) at /<home>/webrtc-pusher/src/rx/socket.c:692
#3  0x0001a758 in server_thread_func (user=0x354130) at /<home>/webrtc-pusher/src/rx/pusher/server.c:317
#4  0xb5573494 in start_thread (arg=0xad7fea90) at pthread_create.c:486
#5  0xb54f6578 in  () at ../sysdeps/unix/sysv/linux/arm/clone.S:73
(gdb) f 2
#2  0x0001d4f8 in rx_socket_can_read (ctx=0x33c0e8, timeoutSec=3, timeoutMicroSec=500000, result=0xad7fe467 "\003\310\005ସ", <incomplete sequence \340\254>) at /<home>/webrtc-pusher/src/rx/socket.c:692
692       r = select(ctx->handle + 1, &readset, NULL, NULL, &timeout);

(gdb) list
687       timeout.tv_usec = timeoutMicroSec;
688       
689       FD_ZERO(&readset);
690       FD_SET(ctx->handle, &readset);
691
692       r = select(ctx->handle + 1, &readset, NULL, NULL, &timeout);
693       
694       if (0 == r) {
695         /* Nope, timed out. */
696         *result = RX_SOCK_TIMEOUT;

Thread 4, KVS-1

(gdb) thread 4
[Switching to thread 4 (Thread 0xa39fca90 (LWP 17408))]
#0  __lll_lock_wait ([email protected]=0xace24980, private=0) at lowlevellock.c:46
46      lowlevellock.c: No such file or directory.

(gdb) bt
#0  0xb557de98 in __lll_lock_wait ([email protected]=0xace24980, private=0) at lowlevellock.c:46
#1  0xb5575f58 in __GI___pthread_mutex_lock (mutex=0xace24980) at pthread_mutex_lock.c:115
#2  0x00063318 in defaultLockMutex (mutex=18446744072315095424) at /<home>/webrtc-pusher/build/build.debug/amazonproducer/src/amazonproducer/dependency/libkvspic/kvspic-src/src/utils/src/Mutex.c:195
#3  0x00043e9c in kinesisVideoStreamDefaultLockMutex (customData=18446744072314954400, mutex=18446744072315095424) at /<home>/webrtc-pusher/build/build.debug/amazonproducer/src/amazonproducer/dependency/libkvspic/kvspic-src/src/client/src/Callbacks.c:40
#4  0x000443c0 in checkIntermittentProducerCallback (timerId=0, currentTime=16452172820799116, customData=18446744072315038536) at /<home>/webrtc-pusher/build/build.debug/amazonproducer/src/amazonproducer/dependency/libkvspic/kvspic-src/src/client/src/Client.c:64
#5  0x00069a8c in timerQueueExecutor (args=0xace1d8b0) at /<home>/webrtc-pusher/build/build.debug/amazonproducer/src/amazonproducer/dependency/libkvspic/kvspic-src/src/utils/src/TimerQueue.c:560
#6  0xb5573494 in start_thread (arg=0xa39fca90) at pthread_create.c:486
#7  0xb54f6578 in  () at ../sysdeps/unix/sysv/linux/arm/clone.S:73

(gdb) f 3
#3  0x00043e9c in kinesisVideoStreamDefaultLockMutex (customData=18446744072314954400, mutex=18446744072315095424) at /<home>/webrtc-pusher/build/build.debug/amazonproducer/src/amazonproducer/dependency/libkvspic/kvspic-src/src/client/src/Callbacks.c:40
40          MUTEX_LOCK(mutex);
(gdb) list
35       * Default lock mutex functionality
36       */
37      VOID kinesisVideoStreamDefaultLockMutex(UINT64 customData, MUTEX mutex)
38      {
39          UNUSED_PARAM(customData);
40          MUTEX_LOCK(mutex);
41      }
42
43      /**

(gdb) f 4
#4  0x000443c0 in checkIntermittentProducerCallback (timerId=0, currentTime=16452172820799116, customData=18446744072315038536) at /<home>/webrtc-pusher/build/build.debug/amazonproducer/src/amazonproducer/dependency/libkvspic/kvspic-src/src/client/src/Client.c:64
64                          pKinesisVideoClient->clientCallbacks.lockMutexFn(pKinesisVideoClient->clientCallbacks.customData, pFrameOrderCoordinator->lock);
(gdb) list
59                      pCurrStream = pKinesisVideoClient->streams[i];
60                      pFrameOrderCoordinator = pCurrStream->pFrameOrderCoordinator;
61                      if (pCurrStream->streamInfo.streamCaps.frameOrderingMode != FRAME_ORDER_MODE_PASS_THROUGH) {
62                          // In the case that frameOrderingMode = FRAME_ORDER_MODE_PASS_THROUGH, pFrameOrderCoordinator is NULL
63                          // so we will segfault, so the check for the frameOrderingMode is important
64                          pKinesisVideoClient->clientCallbacks.lockMutexFn(pKinesisVideoClient->clientCallbacks.customData, pFrameOrderCoordinator->lock);
65                          frameOrderCoordinatorLocked = TRUE;
66                      }
67                      // Lock the Stream
68                      pKinesisVideoClient->clientCallbacks.lockMutexFn(pKinesisVideoClient->clientCallbacks.customData, pCurrStream->base.lock);

Thread 9, KVS-2

(gdb) thread 9
[Switching to thread 9 (Thread 0xa31fba90 (LWP 17413))]
#0  __lll_lock_wait ([email protected]=0xace1d740, private=0) at lowlevellock.c:46
46      lowlevellock.c: No such file or directory.

(gdb) bt
#0  0xb557de98 in __lll_lock_wait ([email protected]=0xace1d740, private=0) at lowlevellock.c:46
#1  0xb5575f58 in __GI___pthread_mutex_lock (mutex=0xace1d740) at pthread_mutex_lock.c:115
#2  0x00063318 in defaultLockMutex (mutex=18446744072315066176) at /<home>/webrtc-pusher/build/build.debug/amazonproducer/src/amazonproducer/dependency/libkvspic/kvspic-src/src/utils/src/Mutex.c:195
#3  0x00043e9c in kinesisVideoStreamDefaultLockMutex (customData=18446744072314954400, mutex=18446744072315066176) at /<home>/webrtc-pusher/build/build.debug/amazonproducer/src/amazonproducer/dependency/libkvspic/kvspic-src/src/client/src/Callbacks.c:40
#4  0x00045678 in getKinesisVideoMetrics (clientHandle=18446744072315038536, pKinesisVideoMetrics=0xa31fb028) at /<home>/webrtc-pusher/build/build.debug/amazonproducer/src/amazonproducer/dependency/libkvspic/kvspic-src/src/client/src/Client.c:459
#5  0x0004f56c in logStreamMetric (pKinesisVideoStream=0xace1e018) at /<home>/webrtc-pusher/build/build.debug/amazonproducer/src/amazonproducer/dependency/libkvspic/kvspic-src/src/client/src/Stream.c:674
#6  0x00050e50 in putFrame (pKinesisVideoStream=0xace1e018, pFrame=0xa1e1b488) at /<home>/webrtc-pusher/build/build.debug/amazonproducer/src/amazonproducer/dependency/libkvspic/kvspic-src/src/client/src/Stream.c:991
#7  0x0004c048 in putEarliestFrame (pKinesisVideoStream=0xace1e018) at /<home>/webrtc-pusher/build/build.debug/amazonproducer/src/amazonproducer/dependency/libkvspic/kvspic-src/src/client/src/FrameOrderCoordinator.c:233
#8  0x0004c4bc in frameOrderCoordinatorPutFrame (pKinesisVideoStream=0xace1e018, pUserFrame=0xa31fb3f8) at /<home>/webrtc-pusher/build/build.debug/amazonproducer/src/amazonproducer/dependency/libkvspic/kvspic-src/src/client/src/FrameOrderCoordinator.c:319
#9  0x00046d14 in putKinesisVideoFrame (streamHandle=18446744072315065456, pFrame=0xa31fb3f8) at /<home>/webrtc-pusher/build/build.debug/amazonproducer/src/amazonproducer/dependency/libkvspic/kvspic-src/src/client/src/Client.c:876
#10 0x00022478 in rx_pusher_send_frame (ctx=0xace00610, frame=0xa31fb3f8) at /<home>/webrtc-pusher/src/rx/pusher.c:465
#11 0x0001b9bc in connection_thread_func (user=0xace005c8) at /<home>/webrtc-pusher/src/rx/pusher/connection.c:354
#12 0xb5573494 in start_thread (arg=0xa31fba90) at pthread_create.c:486
#13 0xb54f6578 in  () at ../sysdeps/unix/sysv/linux/arm/clone.S:73

(gdb) f 2
#2  0x00063318 in defaultLockMutex (mutex=18446744072315066176) at /<home>/webrtc-pusher/build/build.debug/amazonproducer/src/amazonproducer/dependency/libkvspic/kvspic-src/src/utils/src/Mutex.c:195
195         pthread_mutex_lock((pthread_mutex_t*) mutex);
(gdb) list
190         return (MUTEX) pMutex;
191     }
192
193     VOID defaultLockMutex(MUTEX mutex)
194     {
195         pthread_mutex_lock((pthread_mutex_t*) mutex);
196     }
197
198     VOID defaultUnlockMutex(MUTEX mutex)
199     {

(gdb) f 3
#3  0x00043e9c in kinesisVideoStreamDefaultLockMutex (customData=18446744072314954400, mutex=18446744072315066176) at /<home>/webrtc-pusher/build/build.debug/amazonproducer/src/amazonproducer/dependency/libkvspic/kvspic-src/src/client/src/Callbacks.c:40
40          MUTEX_LOCK(mutex);
(gdb) list
35       * Default lock mutex functionality
36       */
37      VOID kinesisVideoStreamDefaultLockMutex(UINT64 customData, MUTEX mutex)
38      {
39          UNUSED_PARAM(customData);
40          MUTEX_LOCK(mutex);
41      }
42

(gdb) f 4
#4  0x00045678 in getKinesisVideoMetrics (clientHandle=18446744072315038536, pKinesisVideoMetrics=0xa31fb028) at /<home>/webrtc-pusher/build/build.debug/amazonproducer/src/amazonproducer/dependency/libkvspic/kvspic-src/src/client/src/Client.c:459
459         pKinesisVideoClient->clientCallbacks.lockMutexFn(pKinesisVideoClient->clientCallbacks.customData, pKinesisVideoClient->base.streamListLock);
(gdb) list
454         pKinesisVideoMetrics->totalTransferRate = 0;
455         pKinesisVideoMetrics->totalFrameRate = 0;
456         pKinesisVideoMetrics->totalElementaryFrameRate = 0;
457
458         // Lock the client streams list lock because we're iterating over
459         pKinesisVideoClient->clientCallbacks.lockMutexFn(pKinesisVideoClient->clientCallbacks.customData, pKinesisVideoClient->base.streamListLock);
460         locked = TRUE;
461
462         for (i = 0; i < pKinesisVideoClient->deviceInfo.streamCount; i++) {
463             if (NULL != pKinesisVideoClient->streams[i]) {

Run 2

Last log:

2022-02-18 21:31:02 DEBUG   postReadCallback(): Wrote 849 bytes to Kinesis Video. Upload stream handle: 0

[22:31:02] debug [   connection.c: 293] |> Start to read.
[22:31:02] debug [   connection.c: 299] <| Read 1451 bytes.
[22:31:02] debug [       pusher.c: 477] |> calling `putKinesisVideoFrame()` with 1421 bytes.
[22:31:02] debug [       pusher.c: 484] <| called `putKinesisVideoFrame()`.
[22:31:02] debug [   connection.c: 293] |> Start to read.
[22:31:02] debug [   connection.c: 299] <| Read 371 bytes.
[22:31:02] debug [       pusher.c: 477] |> calling `putKinesisVideoFrame()` with 341 bytes.
[22:31:02] debug [       pusher.c: 484] <| called `putKinesisVideoFrame()`.
[22:31:02] debug [   connection.c: 293] |> Start to read.
[22:31:02] debug [   connection.c: 299] <| Read 371 bytes.
[22:31:02] debug [       pusher.c: 477] |> calling `putKinesisVideoFrame()` with 341 bytes.
[22:31:02] debug [       pusher.c: 484] <| called `putKinesisVideoFrame()`.
2022-02-18 21:31:02 DEBUG   postReadCallback(): Wrote 2143 bytes to Kinesis Video. Upload stream handle: 0
[22:31:02] debug [   connection.c: 293] |> Start to read.[22:31:02] debug [   connection.c: 299] <| Read 84 bytes.
[22:31:02] debug [       pusher.c: 477] |> calling `putKinesisVideoFrame()` with 54 bytes.
[22:31:02] debug [       pusher.c: 484] <| called `putKinesisVideoFrame()`.
2022-02-18 21:31:02 DEBUG   postReadCallback(): Wrote 775 bytes to Kinesis Video. Upload stream handle: 0
[22:31:02] debug [   connection.c: 293] |> Start to read.
[22:31:02] debug [   connection.c: 299] <| Read 372 bytes.
[22:31:02] debug [       pusher.c: 477] |> calling `putKinesisVideoFrame()` with 342 bytes.
[22:31:02] debug [       pusher.c: 484] <| called `putKinesisVideoFrame()`.
[22:31:02] debug [   connection.c: 293] |> Start to read.
[22:31:02] debug [   connection.c: 299] <| Read 371 bytes.
[22:31:02] debug [       pusher.c: 477] |> calling `putKinesisVideoFrame()` with 341 bytes.
[22:31:02] debug [       pusher.c: 484] <| called `putKinesisVideoFrame()`.
[22:31:02] debug [   connection.c: 293] |> Start to read.
[22:31:02] debug [   connection.c: 299] <| Read 371 bytes.
[22:31:02] debug [       pusher.c: 477] |> calling `putKinesisVideoFrame()` with 341 bytes.
[22:31:02] debug [       pusher.c: 484] <| called `putKinesisVideoFrame()`.
[22:31:02] debug [   connection.c: 293] |> Start to read.
[22:31:02] debug [   connection.c: 299] <| Read 82 bytes.
[22:31:02] debug [       pusher.c: 477] |> calling `putKinesisVideoFrame()` with 52 bytes.
[22:31:02] debug [       pusher.c: 484] <| called `putKinesisVideoFrame()`.
2022-02-18 21:31:02 DEBUG   postReadCallback(): Wrote 774 bytes to Kinesis Video. Upload stream handle: 0
[22:31:02] debug [   connection.c: 293] |> Start to read.
[22:31:02] debug [   connection.c: 299] <| Read 1112 bytes.
[22:31:02] debug [       pusher.c: 477] |> calling `putKinesisVideoFrame()` with 1082 bytes.
[22:31:02] debug [       pusher.c: 484] <| called `putKinesisVideoFrame()`.
2022-02-18 21:31:02 DEBUG   postReadCallback(): Wrote 1449 bytes to Kinesis Video. Upload stream handle: 0
[22:31:02] debug [   connection.c: 293] |> Start to read.
[22:31:02] debug [   connection.c: 299] <| Read 372 bytes.
[22:31:02] debug [       pusher.c: 477] |> calling `putKinesisVideoFrame()` with 342 bytes.
[22:31:02] debug [       pusher.c: 484] <| called `putKinesisVideoFrame()`.
[22:31:02] debug [   connection.c: 293] |> Start to read.
[22:31:02] debug [   connection.c: 299] <| Read 371 bytes.
[22:31:02] debug [       pusher.c: 477] |> calling `putKinesisVideoFrame()` with 341 bytes.
[22:31:02] debug [       pusher.c: 484] <| called `putKinesisVideoFrame()`.
[22:31:02] debug [   connection.c: 293] |> Start to read.
[22:31:02] debug [   connection.c: 299] <| Read 371 bytes.
[22:31:02] debug [       pusher.c: 477] |> calling `putKinesisVideoFrame()` with 341 bytes.
[22:31:02] debug [       pusher.c: 484] <| called `putKinesisVideoFrame()`.
[22:31:02] debug [   connection.c: 293] |> Start to read.
[22:31:02] debug [   connection.c: 299] <| Read 26285 bytes.
[22:31:02] debug [       pusher.c: 477] |> calling `putKinesisVideoFrame()` with 26255 bytes.

2022-02-18 21:31:02 DEBUG   postReadCallback(): Wrote 709 bytes to Kinesis Video. Upload stream handle: 0
2022-02-18 21:31:03 DEBUG   postReadCallback(): Pausing CURL read for upload handle: 0
2022-02-18 21:31:38 WARN    curlCompleteSync(): curl perform failed for url https://s-0cb96ab8.kinesisvideo.eu-west-2.amazonaws.com/putMedia with result Timeout was reached: Operation too slow. Less than 30 bytes/sec transferred the last 30 seconds                            
                                                                       
2022-02-18 21:31:38 WARN    curlCompleteSync(): HTTP Error 0 : Response: (null)

Request URL: https://s-0cb96ab8.kinesisvideo.eu-west-2.amazonaws.com/putMedia
Request Headers:
    Authorization: AWS4-HMAC-SHA256 Credential=<cred>/20220218/eu-west-2/kinesisvideo/aws4_request, SignedHeaders=connection;host;transfer-encoding;user-agent;x-amz-date;x-amzn-fragment-acknowledgment-required;x-amzn-fragment-timecode-type;x-amzn-producer-start-timestamp;x-amzn-stream-name, Signature=<sig>
    connection: keep-alive
    content-type: application/json
    host: s-0cb96ab8.kinesisvideo.eu-west-2.amazonaws.com
    transfer-encoding: chunked
    user-agent: AWS-SDK-KVS/3.0.0 GCC/8.3.0 Linux/5.10.52-v7l+ armv7l
    X-Amz-Date: 20220218T213055Z
    x-amzn-fragment-acknowledgment-required: 1
    x-amzn-fragment-timecode-type: RELATIVE
    x-amzn-producer-start-timestamp: 1645219855.210
    x-amzn-stream-name: roxlu

2022-02-18 21:31:38 DEBUG   putStreamCurlHandler(): Network thread for Kinesis Video stream: roxlu with upload handle: 0 exited. http status: 0
2022-02-18 21:31:38 WARN    putStreamCurlHandler(): Stream with streamHandle 18446744072315065480 uploadHandle 0 has exited without triggering end-of-stream. Service call result: 599                                                                                                                                                                     
2022-02-18 21:31:38 INFO    kinesisVideoStreamTerminated(): Stream 0xfffffffface1d488 terminated upload handle 0 with service call result 599.
2022-02-18 21:31:38 DEBUG   defaultStreamStateTransitionHook(): Stream state machine retry count: 0
2022-02-18 21:31:38 DEBUG   defaultStreamStateTransitionHook():

 KinesisVideoStream base result is [599]. Executing KVS retry handler of retry strategy type [1]
[Thread 0xa27fea90 (LWP 18424) exited]

UPDATE As a test, I've disabled the getKinesisVideoMetrics() by returning STATUS_SUCCESS directly. My app is running for almost an hour w/o any freezes/hangs after this change. I'm still performing a long run test. Even if this might sovle the issue, I'm not entirely sure what happens; if this was a deadlock it would make sense, but from my GDB analysis this doesn't seem like a deadlock.

roxlu
roxlu

@MushMal thanks. The interesting thing is that this issue only happens when I run the code on a RPI. When I run in on my regular dev PC (i7, arch linux) things are fine. It's also happens quite regurarly, after ~30-50 minutes, although I didn't time it exactly.