tcp window scaling, Linux, and broken routers

So, what the hell is TCP window scaling and why do I care?

This had me puzzled for a few days: solarguard.solarcity.com worked fine when I was working from my laptop from home, but didn't when I was at work. That's the Web site the monitoring data from my solar panels is sent to.

At first I thought they don't support path MTU discovery properly (e.g. by blocking most icmp packets, including "fragmentation required"). However, lowering the MTU on my workstation didn't help (but it briefly messed up NFS...)

A co-worker pointed out a the Web site doesn't support Linux, but works fine with Mac and Windows. Bullocks, I thought, silly browser requirements. However, sure enough, even a simple "GET / HTTP/1.0" sent via telnet from Linux doesn't work, but works fine from my Mac laptop. I was stumped.

Until I did these two tcpdumps of a telnet session to solarguard.solarcity.com.

21:09:37.486875 IP bbeck-mac.wlan.lostentry.org.50292 > sol6.solarcity.com.http: S 3232032404:3232032404(0) win 65535 
21:09:37.502028 IP sol6.solarcity.com.http > bbeck-mac.wlan.lostentry.org.50292: S 3190318898:3190318898(0) ack 3232032405 win 16384 
21:09:37.502098 IP bbeck-mac.wlan.lostentry.org.50292 > sol6.solarcity.com.http: . ack 1 win 65535 
21:09:43.279950 IP bbeck-mac.wlan.lostentry.org.50292 > sol6.solarcity.com.http: P 1:17(16) ack 1 win 65535 
21:09:43.476237 IP sol6.solarcity.com.http > bbeck-mac.wlan.lostentry.org.50292: . ack 17 win 65519 
21:09:43.616030 IP bbeck-mac.wlan.lostentry.org.50292 > sol6.solarcity.com.http: P 17:19(2) ack 1 win 65535 
21:09:43.632860 IP sol6.solarcity.com.http > bbeck-mac.wlan.lostentry.org.50292: FP 1:559(558) ack 19 win 65517 
21:09:43.632967 IP bbeck-mac.wlan.lostentry.org.50292 > sol6.solarcity.com.http: . ack 560 win 65535 
21:09:43.633327 IP bbeck-mac.wlan.lostentry.org.50292 > sol6.solarcity.com.http: F 19:19(0) ack 560 win 65535 
21:09:43.650105 IP sol6.solarcity.com.http > bbeck-mac.wlan.lostentry.org.50292: . ack 20 win 65517 

and

21:14:19.460778 IP chef.lostentry.org.56146 > sol6.solarcity.com.http: S 3539793334:3539793334(0) win 5840 
21:14:19.474075 IP sol6.solarcity.com.http > chef.lostentry.org.56146: S 3721548036:3721548036(0) ack 3539793335 win 16384 
21:14:19.474105 IP chef.lostentry.org.56146 > sol6.solarcity.com.http: . ack 1 win 365 
21:14:24.092355 IP chef.lostentry.org.56146 > sol6.solarcity.com.http: P 1:17(16) ack 1 win 365 
21:14:24.238705 IP sol6.solarcity.com.http > chef.lostentry.org.56146: . ack 17 win 65519 
21:14:24.844354 IP chef.lostentry.org.56146 > sol6.solarcity.com.http: P 17:19(2) ack 1 win 365 
21:14:30.238534 IP chef.lostentry.org.56146 > sol6.solarcity.com.http: P 17:19(2) ack 1 win 365 
21:14:30.263687 IP sol6.solarcity.com.http > chef.lostentry.org.56146: . ack 19 win 65517 
21:14:30.263723 IP chef.lostentry.org.56146 > sol6.solarcity.com.http: P 19:21(2) ack 1 win 365 
21:14:30.476289 IP sol6.solarcity.com.http > chef.lostentry.org.56146: . ack 21 win 65515 
21:15:59.668923 IP chef.lostentry.org.56146 > sol6.solarcity.com.http: F 21:21(0) ack 1 win 365 
21:15:59.680890 IP sol6.solarcity.com.http > chef.lostentry.org.56146: R 3721548037:3721548037(0) win 0

What is up with the tcp window size in the Linux dump once the connection is established? 365, vs. 65535 on the Mac. And what does wscale do?

This made me suspicious and a quick visit to our favorite search engine brought up this most excellent blog entry, as well as a somewhat old article on LWN which, however, explains he basic technicalities quite well.

In a nutshell: TCP window scaling is used to work around limitations in TCP that restrict the maximum window size to 64kBytes, by negotiating a multiplier for the window value during session setup. Apparently, there are some broken routers out there that set the wscale option to 0, effectively transparently disabling window scaling without telling anybody. The result are TCP connections that appear to hang for data, but control traffic (like syn/ack, rst, retransmit, etc.) still works!

Sure enough, if I disable TCP window scaling on chef, the SolarGuard website sends the response just fine even in a telnet session, where it didn't work before. Setting the tcp_rmem and tcp_wmem values to the pre-2.6.17 values makes this work properly even with window scaling enabled.

I left window-scaling enabled, but manually set

sysctl -w net.ipv4.tcp_wmem="4096 16384 131072"
sysctl -w net.ipv4.tcp_rmem="4096 87380 174760"

and the Website works fine now.

As an aside, kudos to SolarCity staff. I called Support, they forwarded my problem to someone in Engineering, and I had a sensible answer in my inbox in less than 20 minutes after my initial problem report, followed up with a confirmation that they verified connectivity from multiple places and networks without problems. They even confirmed that my box is working properly.

Maybe they happen to have control over that old router that is not compliant with RFC 1323 from the year 1992, and either fix it, or take it out back and shoot it...

Update:

In Windows the registry key HKLM/tcpip/parameters/Tcp1323Opts is a bit field that controls whether tcp window scaling is enabled (1) or not (0). Setting this to 0 disbles tcp window scaling for both incoming and outgoing connections.

Windows Vista has tcp window scaling enabled by default (so it will have the same problems, that I had with Linux). It can be turned off using the network tuning wizard, or above registry key.

SolarCity SolarGuard

A couple days ago, SolarCity came out and installed the SolarGuard unit. With my Xantrex inverter they can install it straight into the cabling box. It connects via serial port to the inverter and reads the status every few minutes.

The unit itself is actually quite small.


From the outside only the antenna indicates that this inverter is enabled for remote monitoring. I have quite some concerns how long this antenna will last, given that the inverter is in the side-yard of the house and we regularly run trashbins and stuff right by it. To reduce the likelihood of damage to the antenna I turned it side-ways against the recommendation of the technician.


The unit in the inverter communicates with a base station that is connected to my network. The wireless connection doesn't show up when looking for regular wireless networks. I suspect they are using ZigBee (802.15.4) for this link.

Train "Room" Update

The train "room" in the garage is taking shape. The platform frame is done and level. I built this from the faux beams left over from last summer's Asbestos action. The floor is 3/4" plywood supported by 2x4 joists and some additional left-over wood pieces. The whole platform is screwed together, so I can disassemble it easily, if ever necessary.

The final size of the room is 8ft by 8.5ft (240 x 255cm). When I looked at the layout plan I realized that I was working with 240x220cm. I just gained a full 240x40cm of space, which allows me to decompress the whole layout quite a bit.



PDFs: Upper Levels - Lower Level

Underground staging gained a forth track. The ramp to staging got longer, with less grade. I also managed to correct an earlier mistake with where I would place the transition track between K and M tracks.

I angled the Emsingen station at only 15 degress now, which makes for a better flow of track and at the same time moved the station a bit north to have more space in the center of the layout. I could even squeeze in another track to keep cars at the station overnight. The (otherwise flawless) Maerklin track geometry isn't quite accomdating with the 105 degree turn I needed on the West side of Emsingen station, so I will need to fudge a little bit here using flextrack.

Having a deeper northern portion of the layout presents serious access problems, escpecially because I will have track underneath Emsingen station in all the places where one would want to place an easily accessible access hatch. I'm currently planning to make the area around the Emsingen station building removable, so that I can access the north end of the layout.

The tunnel ramp from Emsingen West down the east side of the layout around the mountain has exactly the same grade and location as the respective ramp underneath to staging. I'm planning to build the staging ramp solid enough so that I can simply build the Emsingen tunnel ramp on top it.

I'm not quite sure yet how to arrange trackage in Talheim. Would be nice to connect the two industries in a little bit more prototypical way. I also have a lot of tunnels on this layout, so I want to make sure that the visible track is as conducive to train watching as possible (wide curves in the visible sections as much as possible). There is very little straight track and I might end up scrapping the industry at the south end altogether in favor of building something a bit more interesting near the tunnel entrance to staging, now that there is some more space there.

---

Das Eisenbahn"zimmer" in der Garage nimmt Form an. Der Rahmen fuer die Platform ist fertig und eben. Ich benutze dafuer die alten Deckenbalken von Wohn- und Esszimmer. Der Boden ist aus 1.5cm dicken Tischlerplatten, die von kleineren Balken und diversen Holzresten gestuetzt werden. Die ganze Platform ist geschraubt, nicht genagelt, so dass ich das Ganze wieder einfach auseinander nehmen kann, wenn noetig.

Die endgueltige Groesse des Zimmers ist 240 x 255cm. Als ich mir den Plan fuer die Anlage anschaute, wurde mir klar dass ich die ganze Zeit mit 240x220cm gearbeitet habe, und nun ploetzlich 240x40cm extra hatte, was mir erlaubt die ganze Anlage etwas zu dekomprimieren.

Der Schattenbahnhof bekam ein viertes Gleis. Die Rampe zum Schattenbahnhof wurde laenger mit weniger Steigung. Ich habe auch einen fruehen Fehler korrigiert, so dass das Uebergangsgleis von M-Gleis zu K-Gleis an einer passenden Stelle ist.

Der Bahnhof Emsingen ist jetzt nur noch 15 Grad gedreht, was den Gleisverlauf deutlich verbessert. Gleichzeitig verschob ich den Bahnhof etwas nach Norden, so dass ich in der Mitte der Anlage mehr Platz habe. Es hat noch fuer ein weiteres Abstellgleis gereicht. Die ansonsten einwandfreie Gleisgeometrie der Maerklingleise kommt nicht ganz mit der 105 Grad Kurve in der westlichen Bahnhofeinfahrt zurecht. Ich werde das
mit Flexgleis hinkriegen.

Da das Nordende der Anlage nun deutlich tiefer ist, habe ich ein ernsthaftes Zugangsproblem. Ueberall wo man einen versteckten Zugang anlegen kann, liegen unterirdisch Gleise. Ich plane den Bereich um das Bahnhofsgebaeude Emsingen rausnehmbar zu machen, so dass ich ans noerdliche Ende der Anlage komme.

Die Tunnelrampe von Emsingen West entlang der Ostseite hat genau die gleiche Steigung wie die Rampe zum Schattenbahnhof. Ich plane die Schattenbahnhoframpe stabil genug zu bauen, dass ich die Tunnelrampe einfach oben drauf setzen kann.

Ich bin mir nicht so sicher wie ich die Gleis in Talheim anlege. Waere gut die zwei Industrien etwas realistischer anzuschliessen. Ich habe auch eine ganze Menge Tunnel, und wenig gerade Strecken, auf der Anlage, deshalb werde ich versuchen die sichtbaren Bereiche so anzulegen, dass es Spass macht den Zuegen zuzuschauen (weite Kurven soweit das moeglich ist). Ich kann mir vorstellen die Industrie im Sueden komplett zu streichen und stattdessen die andere Industrie in Talheim etwas interessanter auszubauen.

Solar Energy and Congress

Wow, I still don't really understand they way American Laws are made. According to news reports the Senate amended the housing economic stimulus bill with provisions to extend solar tax benefits. Last year they tried a similar thing which in the end was shot down. I wonder, if they changed the solar tax credit to not have a cap, whether I could claim the difference between the $2000 cap from last year and the 30% on this year's tax return. Would be nice, but I doubt I will be able to do this.

Summer is here!

Friday it was 30+ Celsius. Saturday, as the first real day of summer, saw the heat topping out at 32 Celsius, the hot tub filled with warm water, the day spent outside, children playing on the lawn with the garden hose, and solar production going down, since now the cells warm up substantially enough to be outside their optimal operating range.

Last week with blue skies, but a steady cool wind from the Pacific solar production peaked at 24 kWh/day. On hot days like yesterday we average between 20-22 kWh/day depending on temperature and the angle of the sun.

Wie der Schattenbahnhof funktionieren wird

Find the english version here

Nachdem der erste Teil meiner Bestellung von EuroRail Hobbies & More ankam machte ich mir wieder Gedanken ueber die Anlage. Maerklin 2205 Flexgleis und Uhlenbrock 72600 Dekoder waren. I glaube ich werde nicht mehr Flexgleise brauchen. Muss mal rausbekommen, wie man das Gleis kuerzt. Ich freue mich schon drauf Zuege auf weiten Kurven ueber die Anlage fahren zu sehen ... naja, soweit das bei meinen Platzverhaeltnissen moeglich ist.

Die Dekoder sind sehr klein, und passen genau in den Bereich wo derzeit der Richtungsumschalter ist. Ich warte mit dem Digitalisieren der Loks bis die Intellibox eintrifft, so dass ich die Dekoder gleich ordentlich testen kann. In der Zwischenzeit sollte ich mir eine kleiner SPitze fuer den Loetkolben zulegen...

Da der Schattenbahnhof komplett mit M-Gleisen gebaut werden wird, kann ich mir mit den K-Gleisen aber noch Zeit lassen. Dier erste Phase der Anlage wird praktisch nur aus dem Schattenbahnhof bestehen, inklusive der doppelgleisigen Rampe zur mittleren Ebene und einer Kehre am oberen Ende der Rampe.

Ich werde wohl die Weichen im Schattenbahnhof von Anfang an digital betreiben, so das ich die Fahrstrassenunterstuetzung der Intellibox verwenden kann. Hingegen, plane ich die Weichen im sichtbaren Bereich der Anlage analog zu betreiben. Die Anlage ist klein genug, dass man zu zweit schon gut beschaeftigt sein kann, auch wenn der Schattenbahnhof automatisch per Computer betrieben wird. Da die Rampe nun zweigleisig ist, muss ich nicht warten bis ein Zug ganz im Schattenbahnhof angekommen ist, bis ich einen neuen Zug raufholen kann.

So wird das alles funktionieren: All Schattenbahnhofgleise haben einen Zug. Der Computer waehlt ein Gleis, und schickt den Zug auf die Rampe ins Ausfahrtgleis. Bei Bedarf holt man den Zug vom Ausfahrtsgleis und er faehrt in die Anlage.

Die Weichen im Tunnel sind so gestellt, dass es egal ist auf welchem der zwei Gleise ein Zug in den Tunnel faehrt, er kommt so oder so auf dem Einfahrtsgleis des Schattenbahnhofs an. Es wird ein Hilfsmittel/Signal/Anzeige gebraucht, die anzeigt, ob das Einfahrtsgleis frei oder belegt ist. Vielleicht einfach ein Laempchen, eine rot/gruen Anzeige, oder gar ein Vorsignal. Irgendwas das im Kontext Sinn macht, und auf der Anlage ist. Das Einfahrtsgleis beginnt auf der Rampe und muendet in den Schattenbahnhof. Einfahrende Zuege halten auf dem Einfahrtsgleis kurz vor dem Schattenbahnhof an. Von dort uebernimmt der Computer und waehlt ein freies Gleis, in dem der Zug abgestellt wird.

Damit das alles funktioniert brauche ich drei Belegtmelder. Einer am Anfang des Ausfahrtgleises ("Zug bereit den Schattenbahnhof zu verlassen"), einer am Anfang des Einfahrtgleises (stellt Signal auf "Einfahrtgleis belegt, keine weiteren Zuege nach unten bitte"), einer am Ende des Einfahrtgleises ("hey computer, hier ist ein Zug fuer den Schattenbahnhof"). Wenn dieser Belegtmelder wieder frei meldet, kann er das Signal am Anfang des Einfahrtgleises wieder freischalten. Zusaetzlich kann dort zur Sicherheit eine stromlose Stelle sein, um zu verhindern dass Zuege bei der Einfahrt in the Schattenbahnhof kollidieren. Man kann das auch mit 2 Belegtmeldern machen, wuerde dann aber einiges an Sicherheit verlieren.

Jedes Gleis im Schattenbahnhof braucht eine stromlose Stelle am Ende so dass die Zuege wirklich auch vor der Ausfahrtweiche anhalten und Zusatzgeraete wie Rauchgeneratoren abgeschaltet werden. Theoretisch sollten alle Dekoder damit zurechtkommen.

Order Part 1 and how to operate staging

Today, the first part of an order from EuroRail Hobbies & More arrived. Maerklin 2205 Flextrack and the Uhlenbrock 72600 decoders. It's funny to be in California, unpack the box, and read all the German instruction booklets. I think I don't need any additional flextrack, unless it turns out to be cheaper than regular track of the same total length. Will need to figure out how to properly trim the track to length, as well as how track insulation works with K track. I'm looking forward to watching trains on those nice gentle curves through my scenery ... ok, well, mostly gentle curves I don't have a ton of space to play with.

The decoders are quite small and will fit perfectly in the space where the direction changer switch coil of the locomotives is currently located. I'll wait with the actual digitalization of the locos until the shipment with the Intellibox shows up, so that I can actually test the decoder, too. Meanwhile, I need to get a smaller tip for my
solder iron. The solder pads on those decoders are smaller than I expected...

Since the (hidden) lower level of the layout will be built completely from M track, I still have some time to figure out dealing with K track. The first layout build stage will feature the hidden staging yard and associated ramps only. I will probably digitize the switches on that part of the layout right away to make use of the "Fahrstrassen" support in the Intellibox. I am planning to operate switches on the visible part of the layout manually. Aside from maybe having a computer automatically pick through trains from the staging yard and send them on their way, the layout is small enough to easily keep myself (and another operator) busy with manual operation, especially now that the access ramp to staging has double-track and I don't have to wait for a train to make it all the way down to staging before I can send another train up.

Here's how this might work: All staging tracks are loaded with trains. The computer picks a track sends the train to the outbound track on the ramp. Whenever an operator feels like it, they can release the train from the outbound track into the layout.

The switches in the tunnel to staging are aligned so that no matter which track an operator sends a train to staging, it ends up on the inbound track. There needs to be some visual indication to the operator that the inbound track to staging is occupied. Maybe some green/red light? A distance signal? I'd like something that's on the layout and makes sense in the context. Trains on the inbound track to staging stop before the staging yard and automatic operation takes over. Whenever the computer notices a train stopped on the inbound track, it picks an empty staging track, aligns the switches accordingly and the train proceeds under computer control to the assigned staging track.

For this to work, I need 3 occupancy detectors. One at the top end of the outbound track ("there is a train ready to leave staging") , one at the top end of the inbound track (sets signal "inbound track is busy, no more trains to staging please"), one at the bottom of the inbound track ("yo, computer, send this train to a staging track"). Once that detector shows the inbound track free, it can reset the signal at the top to allow another train in. A safety section can be added to make sure no train can proceed into staging if the inbound track is occupied.

I could do this with 2 occupancy detectors, but would loose a little bit safety and feedback to the human operators, since the wouldn't know when the train actually made it safely into the staging yard.

Each staging yard track needs a computer controlled isolated section at the head end, so that trains really stop before entering the exit switch ladder, as well as any accessories (like smoke generators in steam engines) are turned off. In theory, all decoders should be able to deal with this.