[ part 1 ]
Tonight was the ops session we have been planning for the last few weeks. Since we are in the middle of a global pandemic, we decided to severely limit on-site attendance, institute social distancing, wearing masks, and try out remote participation.
Instead of the usual 10 - 15 session participants in the layout room, we had only four: A yardmaster for Nowheres, and three engineers. Plus myself for the Internet and streaming stuff. We also had three club members joining us remotely as engineers, as well as a remote dispatcher.
Remote control of our web-based layout panels and locomotives on the layout is easily accomplished using a VPN application that connects remote devices to the club network. EngineDriver's automatic discovery of JMRI's Withrottle Server doesn't work over the VPN connection. Instead, remote participants need to enter IP and port number of the computer running JMRI manually. In our case 192.168.8.10, port 12090. The control and dispatcher web panels are available at http://192.168.8.10:3000 over the VPN connection as well.
Over the course of the afternoon I installed cameras in the layout room. I got two Foscam X1 cameras relatively cheap during a Father's Day sale, and used the IP Webcam app on two Android smartphones as additional IP cameras. All four cameras were streaming video into OBS.
I set up the Foscam X1 with the Foscam phone app: Scan the bar code of the camera, enter wifi information, point camera at phone, add camera to phone app for configuration. I was unable to get the browser interface of the camera to work with Chrome, Firefox, or Safari, but didn't try very hard. In Chrome and Firefox I got and empty page, while with Safari I got at least to the login page, but attempts to login resulted in a failed refresh. I didn't debug this in any depth, and haven't tried with a browser on Windows yet.
The OBS setup is quite simple. For each camera:
- Add Media Source and give it a unique name
- Uncheck "Local File"
- Check "Restart playback when source becomes available"
- Set Network Buffering to 1MB
- Input:
- For the Foscam X1 use rtsp://login:password@192.168.8.XXX:88/subVideo
- For the IP Webcam use http://192.168.8.XXX:8080/video
- Leave Input Format empty
- Click [Ok]
The video feed shows up in the projector window. Arrange to liking. I'm using the subVideo feed on the Foscam, since I set the OBS projector to 720p and planned to have 4 cameras on the screen. There is no need to stream HD from the camera if the stream gets downscaled to 360p anyways.
I set up the four cameras to cover as much of the layout room as I could pull off, with the intent to give remote operators as much visibility of the main line as possible, so that they get to see the trains they are running.
OBS can stream to Youtube, which I did for a good part of the session, but the more important feature is to make the projector output visible to the remote participants on Google Meet. I planned to use the OBS VirtualCam plugin, but forgot to restart the browser after installing it, so the OBS Virtual Cam option did not show up as possible inputs for video conferencing, which made me sad. I read that Zoom and Skype had disabled virtual cameras in their applications, so I thought maybe Google had done the same (It didn't. It was just my own stupidity).
I used the Google Meet presentation feature to send the camera window to the video conference instead. Worked fine for me, but not for anybody else. There was a random delay on the window capture presentation stream, between 10 - 120 seconds behind real time. This made it impossible for remote engineers to safely run trains, and put quite a damper on the session. Sorry, guys!
 |
| The control stand |
I spent the evening in front of computers. On the left is the FRS Radio / Meet gateway. This old Dell laptop runs Linux and has separate microphone and head set plugs, with microphones taped to computer and headset speakers. The 17" MacBook Pro in the middle runs OBS and acts as the hub for all the video streams, including broadcasting to Meet and Youtube Live. The 15" MacBook Pro on the right was my control for the Youtube Live stream.
When we do this the next time, I will set up in the dispatchers office, which has a much more comfortable desk area and comfortable chairs.
This was another step to restarting operations at Silicon Valley Lines. Operations sessions as we know them will not be possible for a while. Some remote participation component will be needed for the time begin.
We confirmed that a remote dispatcher with engineers in the layout room is feasible. However, more work is required to reliably set audio levels to avoid clipping, and improve audio quality. At times, engineers in the layout room had trouble hearing dispatcher instructions. Participants on the video conference heard background noise or buzzing that was picked up by the microphone of the FRS gateway computer. These issues should be addressed by replacing the truly analog audio gateway with a circuit that directly connects headset out / microphone in with microphone in / headset out on the radio, taking into account properly adjusting audio levels in both directions.
We need to confirm end-to-end delay of visual feedback through OBS to remote video conference participants is better when using the VirtualCam plug-in.
The club Internet connection has 24 MBit/s downlink and 5 MBit/s uplink. We found that throughput on the uplink was very variable. I stopped the Youtube Live stream halfway through the session after downgrading quality multiple times to free up more bandwidth for Google Meet. That definitely resulted in improved video and audio quality with less artifacts.
IP Webcam did not work reliably on some phones, while it worked just fine on another phone. On problematic phones the app stopped streaming to OBS on a regular basis. Sometimes the app crashed on the phone. I couldn't find a specific root cause for these problems, but noticed that one of the phones I tried got very warm during streaming.
Finally, we need to build a more compelling remote engineer experience. While the camera setup I put together allows for an ok overview of the layout room, it very much has a security camera feel to it. We will try to make it more reliable, but even then it's at best nice as a novelty.
What we really want to do is to put the remote engineer in the middle of the action. If you can't be there, maybe with technology we can provide an experience that is not possible when standing in the layout room.
Stay tuned.
3 comments:
Sounds like you will need to boost the upload speeds on the SVL network a bit. 5 MBPS seems too slow to me to push out all of those videos simultaneously. I get 12 MBPS upload speeds on my home's Comcast service, and with 5 Wireless Access Points scattered around the house and each connected to the router or a network switch via ethernet, the upload speed seems quite stable everywhere I go in-house. Download speeds vary between 40 and 110 MBPS, depending on which computer or iPhone I am using and whether these devices have the internal network hardware to connect with 802.11ac, n, or only g (still in use on the REALLY OLD laptop).
Of course.
However, there's a trade-off here on how much we should spend on Internet that is used only for a few hours per month.
I've read that T-Mobile is now providing 7.5 MBPS upload speeds on their network, and that speed will double, at a minimum, when their 5G network goes active within the next year or so.
Perhaps one of the SVL club members has a T-Mobile account and would allow their smart phone or tablet to be used during your op sessions to serve as an Internet hotspot/tether during those few hours each month. How many gigabytes of data would be transmitted over the hours of such an op session, and would that be too expensive for a club member to accept on their T-Mobile account?
Post a Comment