Copy link Copy Markdown

Member

Refactor API query
Add HLS streams, but prefer websocket streams due to lower latency
Enforce TLS for HLS and websocket streams

@nurupo please test.

Either sideload the plugin from here:
https://raw.githubusercontent.com/bastimeyer/streamlink/plugins/twitcasting/rewrite/src/streamlink/plugins/twitcasting.py
https://streamlink.github.io/cli/plugin-sideloading.html

Or install from my branch via pip:
https://streamlink.github.io/install.html#pypi-package-and-source-code

pip install -U "git+https://github.com/bastimeyer/streamlink.git@plugins/twitcasting/rewrite"

Notes:

The HLS stream URLs always include media=source. I don't know if that's correct, as there's the source field in the JSON response. The websocket streams have different quality options for the different API responses, but the HLS streams do not in the current implementation.
In addition to "fmp4" websocket streams, the API response also includes "llfmp4" streams which the website seems to be using. This is currently not implemented, but this could be changed in another PR. The current websocket streams are still working fine.
Password protected streams are untested. I kept the old implementation for the password option, though. No idea about streams which require a login...
VODs are not supported, but they weren't supported previously either. Looks like VODs are all HLS streams. I didn't want to spend more time trying to figure out the API calls and how the VOD HLS stream URLs are generated.

Copy link Copy Markdown

Was testing on random streams and noticed dvr as yet another option, e.g. on https://twitcasting.tv/nekomata_nyaon

{
  "movie": {
    "id": 762219780,
    "live": true
  },
  "hls": {
    "host": "twitcasting.tv",
    "proto": "https",
    "source": true
  },
  "fmp4": {
    "host": "61-206-39-150.twitcasting.tv",
    "proto": "wss",
    "source": true,
    "mobilesource": true
  },
  "llfmp4": {
    "streams": {
      "main": "wss://61-206-39-150.twitcasting.tv/tc.edge/v1/streams/762219780.672.96/fmp4",
      "mobilesource": "wss://61-206-39-150.twitcasting.tv/tc.edge/v1/streams/762219780.671.96/fmp4",
      "base": "wss://61-206-39-150.twitcasting.tv/tc.edge/v1/streams/762219780.480.64/fmp4"
    }
  },
  "dvr": {
    "hls": "https://dvr171244.twitcasting.tv/tc.dvr/v1/streams/762219780/hls/master.m3u8"
  }
}

If I try to open it directly in a new browser tab (i.e. no Referrer, etc. headers) I get "Forbidden".

Copy link Copy Markdown

Results of some tests.

WebSocket

Seems to work just as fine as it worked before this PR.

Something I have noticed is that for some streams the web player has options for High, Medium and Low qualities, for other just High and Low, and for other there is no quality option as there is only one WebSocket stream available. It looks like they correspond to the llfm4 WebSocket entries in the streamserver.php json reply, with main being high, mobilesource being medium and base being low qualities. Not sure if it makes sense for streamlink to support that?

HLS

As twitcasting_jp stream has ended and it was kind of unique in not using WebSocket, I had to comment out

	if websocket:
	return self._get_streams_websocket(websocket)

part to force it to use HLS on other channels that have both WebSocket and HLS.

HLS seems to be broken:

$ .pyenv-test/bin/streamlink --loglevel=debug 'https://twitcasting.tv/sogaman_piano' best
[cli][debug] OS:         Linux-5.10.0-20-amd64-x86_64-with-glibc2.31
[cli][debug] Python:     3.9.2
[cli][debug] Streamlink: 5.1.1+170.gbecd39d4
[cli][debug] Dependencies:
[cli][debug]  certifi: 2022.12.7
[cli][debug]  isodate: 0.6.1
[cli][debug]  lxml: 4.9.2
[cli][debug]  pycountry: 22.3.5
[cli][debug]  pycryptodome: 3.17
[cli][debug]  PySocks: 1.7.1
[cli][debug]  requests: 2.28.2
[cli][debug]  urllib3: 1.26.15
[cli][debug]  websocket-client: 1.5.1
[cli][debug] Arguments:
[cli][debug]  url=https://twitcasting.tv/sogaman_piano
[cli][debug]  stream=['best']
[cli][debug]  --loglevel=debug
[cli][info] Found matching plugin twitcasting for URL https://twitcasting.tv/sogaman_piano
[utils.l10n][debug] Language code: en_US
[cli][info] Available streams: 3000k (worst, best)
[cli][info] Opening stream: 3000k (hls)
[cli][info] Starting player: /usr/bin/vlc
[stream.hls][debug] Reloading playlist
[cli][debug] Pre-buffering 8192 bytes
[stream.hls][debug] First Sequence: 333; Last Sequence: 336
[stream.hls][debug] Start offset: 0; Duration: None; Start Sequence: 334; End Sequence: None
[stream.hls][debug] Adding segment 334 to queue
[stream.hls][debug] Adding segment 335 to queue
[stream.hls][debug] Adding segment 336 to queue
[stream.hls][debug] Reloading playlist
[stream.hls][debug] Adding segment 337 to queue
[stream.hls][debug] Reloading playlist
[stream.hls][debug] Adding segment 338 to queue
[stream.hls][debug] Adding segment 339 to queue
[stream.hls][debug] Reloading playlist
[stream.hls][debug] Adding segment 340 to queue
[stream.hls][debug] Reloading playlist
[stream.hls][debug] Adding segment 341 to queue
[stream.hls][debug] Reloading playlist
[stream.hls][debug] Adding segment 342 to queue
[stream.hls][debug] Reloading playlist
[stream.hls][debug] Adding segment 343 to queue
[stream.hls][debug] Writing segment 334 to output
[cli.output][debug] Opening subprocess: /usr/bin/vlc --input-title-format https://twitcasting.tv/sogaman_piano -
[stream.hls][debug] Segment 334 complete
[cli][debug] Writing stream to output
[stream.hls][debug] Reloading playlist
[stream.hls][debug] Adding segment 344 to queue
[stream.hls][error] Failed to fetch segment 335: Unable to open URL: https://211-133-247-162.twitcasting.tv/mpegts/762289784-b21bcf716c277453/ts-source-335.ts (404 Client Error: Not Found for url: https://211-133-247-162.twitcasting.tv/mpegts/762289784-b21bcf716c277453/ts-source-335.ts?mode=source&__n=1679264607527)
[stream.hls][error] Failed to fetch segment 336: Unable to open URL: https://211-133-247-162.twitcasting.tv/mpegts/762289784-b21bcf716c277453/ts-source-336.ts (404 Client Error: Not Found for url: https://211-133-247-162.twitcasting.tv/mpegts/762289784-b21bcf716c277453/ts-source-336.ts?mode=source&__n=1679264607527)
[stream.hls][debug] Reloading playlist
[stream.hls][debug] Adding segment 345 to queue
[stream.hls][error] Failed to fetch segment 337: Unable to open URL: https://211-133-247-162.twitcasting.tv/mpegts/762289784-b21bcf716c277453/ts-source-337.ts (404 Client Error: Not Found for url: https://211-133-247-162.twitcasting.tv/mpegts/762289784-b21bcf716c277453/ts-source-337.ts?mode=source&__n=1679264607527)
[stream.hls][debug] Reloading playlist
[stream.hls][debug] Adding segment 346 to queue
[stream.hls][error] Failed to fetch segment 338: Unable to open URL: https://211-133-247-162.twitcasting.tv/mpegts/762289784-b21bcf716c277453/ts-source-338.ts (404 Client Error: Not Found for url: https://211-133-247-162.twitcasting.tv/mpegts/762289784-b21bcf716c277453/ts-source-338.ts?mode=source&__n=1679264607527)
[stream.hls][debug] Reloading playlist
[stream.hls][debug] Adding segment 347 to queue
[stream.hls][error] Failed to fetch segment 339: Unable to open URL: https://211-133-247-162.twitcasting.tv/mpegts/762289784-b21bcf716c277453/ts-source-339.ts (404 Client Error: Not Found for url: https://211-133-247-162.twitcasting.tv/mpegts/762289784-b21bcf716c277453/ts-source-339.ts?mode=source&__n=1679264607527)
[stream.hls][error] Failed to fetch segment 340: Unable to open URL: https://211-133-247-162.twitcasting.tv/mpegts/762289784-b21bcf716c277453/ts-source-340.ts (404 Client Error: Not Found for url: https://211-133-247-162.twitcasting.tv/mpegts/762289784-b21bcf716c277453/ts-source-340.ts?mode=source&__n=1679264607527)
[stream.hls][debug] Reloading playlist
[stream.hls][debug] Adding segment 348 to queue
[stream.hls][error] Failed to fetch segment 341: Unable to open URL: https://211-133-247-162.twitcasting.tv/mpegts/762289784-b21bcf716c277453/ts-source-341.ts (404 Client Error: Not Found for url: https://211-133-247-162.twitcasting.tv/mpegts/762289784-b21bcf716c277453/ts-source-341.ts?mode=source&__n=1679264607527)
[stream.hls][debug] Reloading playlist
[stream.hls][debug] Adding segment 349 to queue
[stream.hls][error] Failed to fetch segment 342: Unable to open URL: https://211-133-247-162.twitcasting.tv/mpegts/762289784-b21bcf716c277453/ts-source-342.ts (404 Client Error: Not Found for url: https://211-133-247-162.twitcasting.tv/mpegts/762289784-b21bcf716c277453/ts-source-342.ts?mode=source&__n=1679264607527)
[stream.hls][debug] Reloading playlist
[stream.hls][debug] Adding segment 350 to queue
[stream.hls][error] Failed to fetch segment 343: Unable to open URL: https://211-133-247-162.twitcasting.tv/mpegts/762289784-b21bcf716c277453/ts-source-343.ts (404 Client Error: Not Found for url: https://211-133-247-162.twitcasting.tv/mpegts/762289784-b21bcf716c277453/ts-source-343.ts?mode=source&__n=1679264607527)
[stream.hls][error] Failed to fetch segment 344: Unable to open URL: https://211-133-247-162.twitcasting.tv/mpegts/762289784-b21bcf716c277453/ts-source-344.ts (404 Client Error: Not Found for url: https://211-133-247-162.twitcasting.tv/mpegts/762289784-b21bcf716c277453/ts-source-344.ts?mode=source&__n=1679264607527)
[stream.hls][debug] Reloading playlist
[stream.hls][debug] Adding segment 351 to queue
^C[stream.segmented][debug] Closing worker thread
[stream.segmented][debug] Closing writer thread
[stream.hls][error] Failed to fetch segment 345: Unable to open URL: https://211-133-247-162.twitcasting.tv/mpegts/762289784-b21bcf716c277453/ts-source-345.ts (404 Client Error: Not Found for url: https://211-133-247-162.twitcasting.tv/mpegts/762289784-b21bcf716c277453/ts-source-345.ts?mode=source&__n=1679264607527)
[cli][info] Stream ended
Interrupted! Exiting...
[cli][info] Closing currently open stream...

Copy link Copy Markdown

Member Author

some streams the web player has options for High, Medium and Low qualities

Yes, this is what the JSON response data refers to in the llfmp4 data. The plugin currently still returns only the highest available quality via the fmp4 websocket stream URL, as you can see in the plugin code. I can change this though with a proper stream quality ranking, so that it outputs all that's available.

HLS seems to be broken:

This looks more like the stream went offline after you started watching. Before I submitted this PR, I tested HLS streams on various live channels and they all worked fine for several minutes, so this also excludes some kind of playlist/segment URL expiration.

I'll check again later.

Copy link Copy Markdown

This looks more like the stream went offline after you started watching.

You are probably right, I'm no longer able to reproduce this, it seems to work now.

By the way, if the stream goes offline, streamlink should terminate instead of trying to download next HLS segments indefinitely, getting HTTP 404.

Copy link Copy Markdown

Member Author

streamlink should terminate instead of trying to download next HLS segments indefinitely, getting HTTP 404.

#2198

- Refactor API query
- Add HLS streams, but prefer websocket streams due to lower latency
- Enforce TLS for HLS and websocket streams

Copy link Copy Markdown

Member Author

The plugin now returns all available websocket and HLS streams, but websocket streams get ranked higher than HLS streams due to latency reasons.

"llfmp4" websocket streams get ignored, because they caused video/audio desync in my tests. Quality-wise, there's no difference and latency-wise there's no difference either from what I could observe. The HLS stream URL from the "dvr" JSON field also gets ignored because of the 403 response.

These changes should be fine now.

Example of a stream with all available qualities for both websocket and HLS streams:

$ streamlink -l debug 'https://twitcasting.tv/gc5r5ogikgv0yvz' best
[cli][debug] OS:         Linux-6.2.7-1-git-x86_64-with-glibc2.37
[cli][debug] Python:     3.11.2
[cli][debug] Streamlink: 5.3.1+49.g9f9e2561
[cli][debug] Dependencies:
[cli][debug]  certifi: 2022.12.7
[cli][debug]  isodate: 0.6.1
[cli][debug]  lxml: 4.9.2
[cli][debug]  pycountry: 22.3.5
[cli][debug]  pycryptodome: 3.17
[cli][debug]  PySocks: 1.7.1
[cli][debug]  requests: 2.28.2
[cli][debug]  urllib3: 1.26.15
[cli][debug]  websocket-client: 1.5.1
[cli][debug] Arguments:
[cli][debug]  url=https://twitcasting.tv/gc5r5ogikgv0yvz
[cli][debug]  stream=['best']
[cli][debug]  --loglevel=debug
[cli][debug]  --player=mpv
[cli][info] Found matching plugin twitcasting for URL https://twitcasting.tv/gc5r5ogikgv0yvz
[utils.l10n][debug] Language code: en_US
[utils.l10n][debug] Language code: en_US
[cli][info] Available streams: 64k (worst), 88k, 220k, 3000k, base, mobilesource, main (best)
[cli][info] Opening stream: main (stream)
[cli][info] Starting player: mpv
[plugin.api.websocket][debug] Connecting to: wss://115-69-195-15.twitcasting.tv/ws.app/stream/762322563/fmp4/bd/1/1500?mode=main
[cli][debug] Pre-buffering 8192 bytes
[plugin.api.websocket][debug] Connected: wss://115-69-195-15.twitcasting.tv/ws.app/stream/762322563/fmp4/bd/1/1500?mode=main
[cli.output][debug] Opening subprocess: mpv --force-media-title=https://twitcasting.tv/gc5r5ogikgv0yvz -
[cli][debug] Writing stream to output
[cli][info] Player closed
[plugin.api.websocket][debug] Closed: wss://115-69-195-15.twitcasting.tv/ws.app/stream/762322563/fmp4/bd/1/1500?mode=main
[cli][info] Stream ended
[cli][info] Closing currently open stream...

$ streamlink -l debug 'https://twitcasting.tv/gc5r5ogikgv0yvz' 3000k
[cli][debug] OS:         Linux-6.2.7-1-git-x86_64-with-glibc2.37
[cli][debug] Python:     3.11.2
[cli][debug] Streamlink: 5.3.1+49.g9f9e2561
[cli][debug] Dependencies:
[cli][debug]  certifi: 2022.12.7
[cli][debug]  isodate: 0.6.1
[cli][debug]  lxml: 4.9.2
[cli][debug]  pycountry: 22.3.5
[cli][debug]  pycryptodome: 3.17
[cli][debug]  PySocks: 1.7.1
[cli][debug]  requests: 2.28.2
[cli][debug]  urllib3: 1.26.15
[cli][debug]  websocket-client: 1.5.1
[cli][debug] Arguments:
[cli][debug]  url=https://twitcasting.tv/gc5r5ogikgv0yvz
[cli][debug]  stream=['3000k']
[cli][debug]  --loglevel=debug
[cli][debug]  --player=mpv
[cli][info] Found matching plugin twitcasting for URL https://twitcasting.tv/gc5r5ogikgv0yvz
[utils.l10n][debug] Language code: en_US
[utils.l10n][debug] Language code: en_US
[cli][info] Available streams: 64k (worst), 88k, 220k, 3000k, base, mobilesource, main (best)
[cli][info] Opening stream: 3000k (hls)
[cli][info] Starting player: mpv
[stream.hls][debug] Reloading playlist
[cli][debug] Pre-buffering 8192 bytes
[stream.hls][debug] First Sequence: 149; Last Sequence: 152
[stream.hls][debug] Start offset: 0; Duration: None; Start Sequence: 150; End Sequence: None
[stream.hls][debug] Adding segment 150 to queue
[stream.hls][debug] Adding segment 151 to queue
[stream.hls][debug] Adding segment 152 to queue
[stream.hls][debug] Writing segment 150 to output
[stream.hls][debug] Segment 150 complete
[cli.output][debug] Opening subprocess: mpv --force-media-title=https://twitcasting.tv/gc5r5ogikgv0yvz -
[stream.hls][debug] Reloading playlist
[cli][debug] Writing stream to output
[stream.hls][debug] Writing segment 151 to output
[stream.hls][debug] Segment 151 complete
[stream.hls][debug] Adding segment 153 to queue
[stream.hls][debug] Writing segment 152 to output
[stream.hls][debug] Segment 152 complete
[stream.hls][debug] Writing segment 153 to output
[stream.hls][debug] Segment 153 complete
[cli][info] Player closed
[stream.segmented][debug] Closing worker thread
[stream.segmented][debug] Closing writer thread
[cli][info] Stream ended
[cli][info] Closing currently open stream...

Closed

4 tasks

Closed

5 tasks