SVL: Fall Open House

Remotely controlled train 4837 entering Kaos Jct on a green signal

The Silicon Valley Lines Fall Open House is in the box. I had a lot of fun, but as such events go, it was also stressful. The video-enabled Webthrottle worked well for remote visitors, once we got over a major mistake I made when sending the throttle URLs to the first visitor group.

After that I got more into a rhythm: Send emails with personalized throttle links to confirmed visitors, have them join on the video conference, give introduction talk, hand off to train master, and while the session is running work on the next batch of emails.

At the Audio/Video control stand preparing email notifications for the next session (Photo Credit : Josh)

We had planned for running a round-trip session every 20 minutes. That was a bit too aggressive, since it takes a train about 15 minutes to drive the Open House loop. We had pre-staged trains around the layout, so 3 visitors could start with running their train right away. However, 5 minutes was not enough to get operators on the video conference, familiarize them with the throttle control, and oriented on the layout.  Including an occasional break for the crew would have been nice, too. When we do this again, we should clock the departures on half hour increments, but otherwise can use the same process. 

I lost all audio output on the main computer controlling the Youtube Live stream for a while, and couldn't get the audio working again, so I substituted with a room microphone. However, audio quality on that microphone was very poor. Now -- with some distance -- I'm thinking I should have restarted the computer at the cost of dropping the live stream for a while. That would have been better than having the live stream with no usable audio.

Train 161 rounding the curve at Bayshore

The crew in the layout room was great. The cameras and the trains worked really well. The DSL connection held up well with only minimal drops affecting quality, even though we had the upstream bandwidth maxed out at times. 

We learned a lot during this Open House, which was probably the first time ever multiple remote operators ran trains on a club layout in parallel with individual video feedback. We have a long list of things we can improve on, but we'll get to that another time.

As we shut down the layout in the afternoon and closed up shop, everyone was tired, but happy. I got only positive feedback from visitors during and after their sessions. This was a long day, but totally worth it.

SVL: Rebuilding Camera 3

Back in September, as we were gearing up for another full-blown socially-distanced ops session at Silicon Valley Lines, I quickly needed to build another camera car. I had already built a supplementary power supply for James' Dash9 dummy locomotive using a well car and a 10,000 mAh USB power bank. I wanted a self-contained camera car, so I decided to build a cradle for the second power bank I had, and mount the camera on the side of the power bank. That setup became Cam3.

This worked, and we used the car in ops sessions since. However, over time it became obvious that for some reason this car had worse wifi reception than Cam4, my original camera design. That's very curious since it's basically the same hardware ... with two exceptions: Cam4 is powered with a JuiceBoxZero connected to a Li-Ion battery, and Cam3 hugs a big hunk of metal. Side by side comparisons showed that Cam3 performed clearly worse than Cam4, even after reducing the network data requirements of the video streams as can be seen in the SVL operator training video.

After acquiring a power bank in a smaller form factor, as well as a low-profile, right-angle Micro-USB plug I was ready to replace the first design for mounting Cam3. I decided to install the PiZero horizontally, but since the USB plug is sticking out quite a bit, I mounted the PiZero on a styrene sheet and built a support structure into the well car with the PiZero mount sticking out about 1/4" on both sides of the car.  I verified with an NMRA gauge that the wider platform is still within the loading gauge. Next I built a camera mount over the board and was very satisfied with my work. Here is Cam3 on the well car next to Cam4 ...

That satisfaction lasted for about 5 minutes, when I remembered that we have a couple signal masts in Jacksonville that are mounted between the tracks and clearly this arrangement is not going to fit.

It was time for design number three, which I call "The Fin".

I rotated the camera 90 degrees, so that it stands on its side. The angled Micro USB plug is now at the top and the USB cable threaded through the support structure.

I was careful to make sure that the camera is well behind the lead truck. I needed to rotate the camera image, which is easily accomplished by passing -rot 90 to mjpg_streamer. I'd still like to find an actual low-profile Micro-USB plug. There are some variants of what I'm thinking of available from Chinese suppliers (e.g. reasonably cheap on aliexpress.com, or very expensive on amazon.com), but they all have lead times of about 4 weeks and the plugs look very finicky and prone to failure. It's probably easier, and definitely cheaper, to just solder the charging leads directly to the PiZero.

In either case, now that I liberated Cam3 from the hunk of metal, it seems to performs much more smoothly.

SVL: Fixing CabCam video throughput issues

 

Last Friday I went to the club layout and tested the Web Throttle code. The code worked, but video quality was terrible. As soon as I tried to get two video feeds off the camera to WebThrottle and OBS respectively, the video streams produced only one image every 10 seconds at best. That's clearly not acceptable for what we are trying to do here.

I experimented with camera settings over the weekend at home. My theory was that due to the difficult Wifi conditions in the layout room, we're simply not getting the network throughput we need to get stable video streams. When running two streams at home I found that the PiZero kept a network send queue of more than two Megabyte or more, i.e. mjpg_streamer is stuffing bytes into the network queue, but they are not making it off the device fast enough. Eventually, buffers will overflow, the network code backs off, the queue drains out, and we see skipped video. In tunnel sections on the Welztalbahn, the queue dropped to about one Megabyte. My tunnels are very dark, so the jpgs produced by mjpg_streamer are much smaller and the network can catch up. In addition, the PiZero CPU was only about 30% idle when processing camera data to two streams when a train is moving. Not a lot of headroom. As a consequence, while the video works ok, even at home it was a bit choppy.

So with two streams we are simply pushing too much data into the network layer that can't get drained out fast enough over the network. The solution is to push less data. I reduced the video resolution from 640x480@15fps to 480x360@10fps, which reduces the raw bandwidth need to about one third. Testing on the Welztalbahn yielded a very smooth and stable video stream.

Sunday evening I was back at SVL and tested my theory at the layout. A single WebThrottle stream to my smartphone was smooth and stable, even in the Cavenaugh area that is particularly problematic. Next I pulled out the laptop with OBS and streamed the camera feed to Youtube while working on mounting location signs for the upcoming Open House. While before we often had a lag of a bit less than 5 seconds when watching the camera stream, video now was almost realtime and so smooth that you can see the flat car sway back and forth on the rails.

Towards the end of the evening, I pulled out camera 3 which is known to have reception issues because of how the PiZero sits right next to the large USB power bank. With the adjusted settings, quality is better, but still visibly worse than camera 4. You can see the difference side by side starting around 1:08:00  in the work video. That needs to be get fixed before Saturday.

I used the work video source to make an instructional video for Open House operators showing how to use the WebThrottle and what the location signs look like.

NMRAx: "Remote Operations at Silicon Valley Lines"

Today Dave Falkenburg and I presented a clinic on "Remote Operations at Silicon Valley Lines" at the NMRAx Virtual Convention. The session was live-streamed on Facebook and Youtube.

Regular readers of this blog will recognize some of the content of the presentation. There's plenty of new content as well, so I hope it was interesting in either case.

Here is the raw recording of the presentation of today's NMRAx linked to the beginning of our talk.

TSG Multimedia: Remote Operating Session Concepts

I've been waiting for a couple weeks now to be able to write this post. John Abatecola from TSG multimedia visited Silicon Valley Lines in August and filmed segments for this video before and during an operations session at Silicon Valley Lines. I explain various aspects of how we add remote operators into operating sessions at SVL, including an overview of the hardware, software, and systems setup, as well as pitfalls. This allows us to minimize the number of people in the layout room to enable social distancing.

Ops Prep

I've decided to build a second camera car for Silicon Valley Lines Ops sessions, so that we have a spare camera in case there are problems. For this one I wanted to try and use the existing USB battery holder I built last month and mount the Raspberry Pi on the side of the battery. I tacked the Raspberry Pi Zero to a thin sheet of styrene that can slide between the battery and the styrene supports. I built a support structure for the camera that's glued to the styrene sheet the Pi sits on, and fits between battery and car side. Then I carefully twisted and bent the camera cable to make a 90 degree turn so that it it oriented correctly. I could turn the image in software, but prefer to go with physically correct orientation from the start if possible. After taking the photos below, I added another styrene sheet to protect the camera cable from bumps.

This setup must be placed in a container well car because it is quite high and completely fills out the modern HO loading gauge. In the foreground is my prototype camera setup on an old Maerklin low side-boards car.

I also built another battery holder that we can use for James' dummy locomotive. This is a very simple build made from styrene sheet and strips with some triangular support pieces. A quick and easy after dinner exercise.

And here's the end result with battery placed in a modern container well car. There's no need to secure this load because the battery is so heavy that it's held in place merely by friction.

SVL: Broadcasting Trains

After the business meeting tonight we tested running trains with remote operators and two CabCams. For some time we also streamed the to Youtube Live and club members are discussing the experience. We had one camera run out of battery just before the live-stream started, so most of the stream is the cab view from a single train.

My camera ahead of the train in the video.

James mounted his camera in a dummy unit. Looking good!

SVL: Looking Around The Corner

[ CabCam posts ]

When mounting a camera on a flat car for layout videos or remote operators, a common approach is to set the camera close to the front of the flat car, since we want to give the illusion of being in a locomotive cab. Seeing the front of the flat car destroys that illusion.

However, doing this has an unfortunate side-effect: Depending on the length of the flat car used viewers now complain that they can't see as much of the scenery along the track as they had hoped. Especially in tighter curves there is a distinct lack of scenery visible on the inside of the curve.

The camera doesn't "look around the corner".

Part of the reason for this is basic geometry: The car rides on a curve. The center of the camera view is on a tangent off the center of the curved track. Hence, the center line of the camera image and the center line of the track diverge. The tighter the curve, the larger the divergence, the more scenery on the outside of the curve is taking up screen real estate on the inner side of the camera view.

An additional factor is the length of the flat car. The longer the car, the more the ends of the car hang to the outside of the track center line, making the natural tangent effect worse.

The effect is demonstrated by the car in the middle above. The teal triangle simulates the view of a camera mounted at the very front of the car looking straight ahead. Barely anything to the right of the track is is visible to the camera, while plenty of scenery to the left of the track can be admired.

There are multiple ways to even this out:

• Mount the camera in or near the front, but use a short flat car: By reducing the overhang at the end of the car, the camera's lens is positioned closer to the tangent off the track center line. Positioning the camera above the center of the front truck, or front axle, minimizes the overhang.
• Using a long flat car, mount the camera in the middle of the car: Long cars have an overhang to the outside of the curve at the car ends, and an overhang to the inside of the curve in the middle of the car. We can use the middle overhang to move the center line of the camera to the inside of the curved track center line and pick up more visibility.
• Using a long flat car, mount the camera in the rear of the car: Moving the camera further towards the rear of the car creates a "steering effect" where the front truck is "turning the car deck into the curve." In this setup the camera center line is not on a tangent off the curve, but on a chord. Hence the camera view captures much more of the scenery to the inside of the curve as demonstrated by the car on the right in the image above.
• Using a wide-angle or fish-eye camera lens can help as well, but results in distortions along the edges of the image.

Using a single long flat car is convenient, since it has plenty of space for the camera and supporting gear like batteries.

If too much of the front half of the car is visible in the resulting video, in many cases cropping the video can hide the car deck. If that's not possible or insufficient, push the camera lens forward until the car deck is no longer visible. This will still produce a better video experience than mounting the camera at the very front of the car. Alternatively, use two flat cars. The front car holds the camera pushed as far to the rear as possible, and the second flat car holds the support gear.

SVL: Club evening with remote operator

[ CabCam posts ]

For tonight's club meeting I brought the camera to the layout, so that we could test more of the applicability of this on the actual layout. It worked well.

Here is James running the train on the layout from his home in the Santa Cruz Mountains.

The camera opens up new perspectives for ops. Here's a screenshot taken by another member participating in the video conference.

This time I set up in a much more comfortable location in the layout room.

SVL: Raspberry Pi Zero Camera Car with Streaming Video

[ CabCam posts ]

We're still tuning the software and streaming setup, but the hardware looks feasible.

This is a Raspberry Pi Zero W with the Zero Spy Cam and a Juicebox Zero battery board connected to a 2000mAh Li-Ion battery. A completely self-contained camera setup that easily fits on a HO flat car and can stream video over existing Wifi. If I stack the components more intelligently, it fits on a much shorter car, too.

During the Show&Tell at tonight's club meeting, we demonstrated the setup at the workbench and on the Welztalbahn.

Members on the video conference were able to observe the trip with only a few seconds delay. This worked much better than expected.

So far, so good. The real test will be trying this out at the club.