Work machine saturating network - solutions?

[crombie]

So I ran into an issue this morning where some random update on my work machine saturated our connection. Naturally I went poking around and of course Deco does not have 'true' QoS. Technically I can set devices as priority except for the work device, but our phones randomize MAC and I see them constantly being added as 'new devices'.

I saw suggested the possibility of using SQM through OpenWRT, but all I have are 32 bit machines which will not run SQM as it requires 64bit.

NOW I had thought that OpenWrt being the wi-fi point with only 2.4GHz access would limit the potential draw but I guess the connection is technically 'n'. So in theory faster than our internet connection. (300Mbps down / 10 Mbps up)

So am I basically out of options now? Especially since I have no idea what the culprit was on the work machine, but multiple people were experiencing the same issue.

I mean, I have the same problem with torrents or Steam and all I needed to do was set their download to a reasonable percentage of our connection.

 

[crombie]

Unfortunately there is no option for more than 10Mbps until they get fibre to our area. BUT I figured out why sqm was refusing to install, it is because the OpenWrt laptop was setup in AP mode. I reset it to router, and now all the shaping is in place. I limited the max connection to 150Mbps and 8Mbps which in theory should resolve the issues. The laptop can choke itself for all that I care, I just want the other machines on the network to have the bandwidth too!

 

[crombie]

You're talking about a normal windows or other OS update taking all 300 megabit download, and/or possibly using up the 10 megabit upload in 'acknowledgement' traffic?

Honestly, it is unknown, if it happens again me and my other team will be sending a ticket in because not everyone has tech knowledge to try to setup this solution. I know Crowdstrike kept pinning out on the CPU, but it was mainly a Windows component that was taking the bulk if the Network in Task Manager.

If so, you could set the ethernet/network adapter on the machine to limit its autonegotiation to only 10/100 instead of 10/100/1000 and your downloads will slow down a good bit, which should also free up the upload a bit. If you ever need to download something extra fast later, just put it back to full auto.

Oooh, that is also a great idea! I will see if I have access to that on the machine. And if not I will have a ticket to have that updated.

Sounds like it might be a device on wifi though, if that is the case, you can have it limited to 2.4 ghz if you want, or like you seem to have done, change the AP to a router mode and use a fair queuing control that should help manage things a bit more automatically, though it sounds like you might now have double routers which can cause double NAT. That sometimes causes issues of its own.

The affected devices were a mix of hard-wired and wifi, the device causing the issues was direct wired to the Deco X60 we have. It is no longer, it is on the 2.4GHz wifi I setup on OpenWrt for the time being. As far as I can tell it isn't double-NAT. The path should be:

Internet -> Deco X60 (DHCP) -> Everything Else -> NB200 (OpenWrt) on one route, and then NB200 -> Wireless on another. I have not seen any of the other devices dropping off of the Deco getting an IP in that range.

 

[crombie]

Yeah, I am trying not to spend money or too much money on a solution. What is funny is I literally just ditched a box of routers when we moved, and one even had OpenWrt installed. It would have been the perfect solution for this issue with no fiddling. I figured with moving my Plex to Mint that this OpenWrt on the Atom would be cake to setup. It has been anything but, and it is even looking like QoS might not work properly.

NOW that said, over the 2.4GHz connection that is being limited to 40Mbps/s download which my basic networking understanding means it could still be saturated with ACK, but from a raw bandwidth perspective it is a small portion of the total.

Honestly, if this doesn't work out, and if it happens again, I will just turn off wifi and/or pop the Ethernet out of the computer and submit a ticket. If this were my personal machine you 100% know I would hunt the service or app down to the ends of the Earth. For a work computer, that is really a 'them' issue to figure it out.

 

[crombie]

you send the update from the DP to a client and then that client can send it to other clients to lessen the load on the P2P link between offices

You know, this actually seems to align with what was happening. It was like P2P was saturating our connection. Come to think of it they did recently engage with a service that is supposed to automate app updates.

 

[crombie]

Well, would you look at that, last night just past my work hours our connection started to go down again. Pulled the ethernet from the OpenWRT device and a few moments later the network was back to normal.

I researched the app they started using for updates, and it does have a P2P component. I am curious if they just recently decided to turn that on, and that would likely mean it is my upload getting saturated.

Ticket submitted, but my fix is to pull the ethernet from the OpenWrt device if it happens again until or if I manage to get QoS working on that device which I am not holding out much hope that it can be done.

 

[crombie]

Sadly no dice on the ticket, but I got qos scripts working on the OpenWrt device!

 

[crombie]

I mean, we have two real providers here, Bell (fibre) and Eastlink (cable). All resellers are through Eastlink. And that is limited to the 10Mb upload (except Purple Cow which is doing a very limited fibre rollout).

I was with Bell for years, but could no longer justify paying over $200 a month for Internet even with work paying half of that. Currently work pays for 100% of the internet connection, and if the issue has happened again I have not noticed it since that machine is throttled.

And the side benefit is since it is on it's own subnet it no longer can discover any network shares that I turn on to move files between my home computers.