Why is that surfcam offline?

get.coned Wednesday, 29 Dec 2021 at 01:29 pm new

Thanks for adding an Alex bluff cam. No more zooming in on the Alex Headland to check it!

I've noticed it hasn't been added to the surfcams list on the main menu and can only be accessed from the nearby cams section when looking at another cam.

I'm trying to add it to my multi-cam but can't without it being on the list.

Just checking if this is an oversight or is already on the to-do list?

Cheers

Suitsurfer Tuesday, 11 Jan 2022 at 12:43 pm new

Any word on when the Lorne and 13th Beach web cams will be back online?

Cheers

thermalben Tuesday, 11 Jan 2022 at 01:29 pm new

Lorne UPS battery died on the weekend. I immediately shipped down a replacement but just got an email this morning that "Unfortunately, our current stock is having some quality issues, hence we have stopped dispatching the old stock. We have now processed full refund for you."

Eh? Anyway, gotta find another one and send down to get swapped over.

13th Beach is still awaiting the surf club to be finished being rebuilt.

thermalben Monday, 17 Jan 2022 at 07:22 am new

Just an update on Lorne. Here's a timeline of what we've done so far:

Our monitoring system detected the cam went down at 8:20pm on Jan 5th. The nature of the outage suggested a power issue (rather than internet outage) but we needed someone on site to visually confirm, so that we knew what equipment needed to be replaced.

Contacted someone at the surf club on the morning of Friday Jan 6th, but they were delayed in gaining access until Sunday 8th. Confirmed the most likely source of the problem was the battery in the UPS had died.

Usual source of replacement batteries is JayCar. Nearest store is Geelong, so went down the postage route - but after an hour of trying, was unable to get one sent down - initially they 'wouldn't post to a PO Box'; then they just simply wouldn't post at all (maybe because it's a battery, i.e. restricted freight item? I dunno).

So, I tried another ten online retailers - keeping in mind, this ain't AA Energizers I'm looking for - UPS batteries are only found in a few speciality stores, and whilst there are some after-market batteries, it's important that they have the right dimensions, as it's a tight fit in the back of the UPS.

Finally found one through a third party reseller on Kogan. Paid, booked etc, locked in - but wasn't sure how long it's take to deliver - Lorne's a small country town, and there are massive delivery delays at the moment. Date is still Sunday Jan 8th.

Tues Jan 11th, received an email from the seller. "Thanks for purchasing this item. Unfortunately, our current stock is having some quality issues, hence we have stopped dispatching the old stock. We have now processed full refund for you."

WTF? OK, back to square one. Instead of buying a replacement battery, I decided to purchase a whole replacement UPS. Got one within an hour (most suppliers have zero stock at the moment), placed an order. Supplier is a regular source of hardware for us, so expecting reliable service. They "had stock on hand" so assuming they dispatch within 24 hours, will hopefully be delivered by Friday Jan 14th - though most likely earlier in the following week (i.e. this week).

Yesterday, having not received notification of the delivery, I checked the order status and it's "currently pending".

WTF? Funds have left the bank account, their website said stock was on hand. Have emailed asking what is up with the order.

So that's where we're at with Lorne. Just another frustrating day in the life of surfcam maintenance.

GuySmiley Monday, 17 Jan 2022 at 08:51 am new

Why not contact local lad VicLocal to pick up from JayCar in Geelong & deliver to Lorne Ben, if he refuses threaten to ban him from the forums. Haha. He sounds like a community minded citizen. What do say VL?

Cockee Monday, 17 Jan 2022 at 09:20 am new

Bit of a Catch 22 there GS - VL knows he can't afford to take off more than a few mins of his day from educating us on the forums, but if he doesn't give up a slice of his valuable teaching time, Ben will ban him. Indeed a complex and onerous proposition for VL to consider.

goofyfoot Monday, 17 Jan 2022 at 10:34 am new

VL reply coming shortly - “so here’s the thing you blokes’…”

Vic Local Monday, 17 Jan 2022 at 10:45 am new

Just got boosted and waiting my 15 mins in north geelong. Happy to drop by specialty electric store in geelong and head to Lorne.

GuySmiley Monday, 17 Jan 2022 at 11:31 am new

Good one Viclocal that’s the spirit well done

Mmmm, he started his SN career running an errand for a desperate Ben but quickly progressed to Vicco reporter before moving up to ... ?

Just my attempt of humour in an otherwise dull onshore day

0-D43-A759-76-C0-4909-BEA8-6-D9-AE084-B58-F
actress foch crossword

Vic Local Monday, 17 Jan 2022 at 11:51 am new

GuySmiley wrote:

Good one Viclocal that’s the spirit well done

Mmmm, he started his SN career running an errand for a desperate Ben but quickly progressed to Vicco reporter before moving up to ... ?

Just my attempt of humour in an otherwise dull onshore day

0-D43-A759-76-C0-4909-BEA8-6-D9-AE084-B58-F
actress foch crossword

About 15 years ago I stepped in to replace the regular Surf Coast reporter. It was a week on northerly winds and mid size swell. I kept saying PI and Mornington were better value for Melbourne surfers. Those reporters wised up and started saying Surf Coast reefs would be good value.
Ahh good times.

GuySmiley Monday, 17 Jan 2022 at 12:45 pm new

Nice VL, exactly as I’d play it. Pick up the batteries?

thermalben Monday, 17 Jan 2022 at 04:34 pm new

Thanks for the offer VL, however I've decided to change tact - I'm gonna remove the UPS from the equation (not quite as easy as it sounds, but a better long term solution IMO).

thermalben Monday, 17 Jan 2022 at 06:55 pm new

Tis back.

dannyz Tuesday, 18 Jan 2022 at 11:45 am new

Any idea when we’re getting the Avoca cam back online? it’s been ages now guys

thermalben Tuesday, 18 Jan 2022 at 12:06 pm new

Hopefully in the next few days. First we had a tech issue, then a delivery delay, then the Xmas/New Year holiday break (where no-one was available) followed by a Covid-19 close-contact for our technician (requiring yet another reschedule) - all of which prevented us from getting this swapped over in a timely manner.

thermalben Wednesday, 19 Jan 2022 at 12:27 pm new

Avoca is back.

dannyz Thursday, 20 Jan 2022 at 01:11 pm new

awesome thanks mate looks great!
except it’s not planning anymore?

thermalben Thursday, 20 Jan 2022 at 04:41 pm new

Haven't quite set that up yet.. in the next day or so

dannyz Friday, 21 Jan 2022 at 02:07 pm new

ahh sweet thanks mate.. can i request that it stops on the current position when you do get it panning? it never used to, it would pan right past this current shot.. it’s a great part of the beach

dannyz Thursday, 3 Feb 2022 at 09:41 am new

thermalben wrote:

Haven't quite set that up yet.. in the next day or so

hey mate any update on the panning? i love the current position it’s in but wanna see the rest of the beach too.. yeww

epictard Wednesday, 23 Feb 2022 at 08:12 am new

Wollongong has been down for a week or so now.
I am just checking in to see when you think it'll be coming back online?

thermalben Wednesday, 23 Feb 2022 at 11:02 am new

Had a technician out there last week, confirmed the camera is dead (ironically, had a technician out the week before swapping out the modem and other hardware, though didn't touch the camera.. just a coincidence I suppose).

Have sent down replacement hardware, will be installed ASAP.

Kered Ausden Wednesday, 23 Feb 2022 at 03:14 pm new

Margaret River has been down for a week or so now.
I am just checking in to see when you think it'll be coming back online?

bluediamond Wednesday, 23 Feb 2022 at 03:53 pm new

Would be interested in Margs cam too. The Albany buoy has been out lost at sea a few months ago, so currently can only go by the Cape Naturaliste buoy (a different ocean) and the Esperance buoy, 400kms down the coast, by which time the swells already passed here. The Yals cam doesn't give a true reading of open ocean swell.
Unless i'm looking at the actual waves there's zero readings to go by from home. Except when the black cockatoos fly sideways and the Pelicans start dancing on the rocks in the inlet ;-)

thermalben Wednesday, 23 Feb 2022 at 03:55 pm new

Same deal with Margs, have sent over some replacement hardware (camera finally gave up the ghost) though we're swapping it out with new gear so it's a bigger project - have been waiting for borders to re-open so I can send over a technician but am still not entirely sure on entry restrictions.

WA cams have been a major headache to maintain over the last 2 years because of travel restrictions. Looking forward to upgrading 'em all over the next couple of months.

bluediamond Thursday, 24 Feb 2022 at 07:15 pm new

Sweet as. Cheers Ben. Will add that to my list of McGowan....ahh never mind. :-P

thermalben Friday, 25 Feb 2022 at 09:24 am new

Lorne and Fairhaven are offline, thanks to a faulty software patch.

Last year, NBN announced they were implement two new functions to VDSL2 lines, Save Our Showtime (SOS) and Robust Overhead Channel (ROC), designed to improve the stability of an internet connection.

We were told that modems needed to be compatible by early March, so yesterday afternoon we started rolling out firmware updates (provided by our hardware manufacturers).

All of the updates worked fine, except Lorne and Fairhaven, which were both using the same kind of modem (but different to the rest of the network). As such, these modems now appear to be bricked, and will require a site visit to rectify. We'll try to get that done early next week.

Apologies for the inconvenience.

eel Friday, 25 Feb 2022 at 01:15 pm new

So the Narrabeen cam?? Been out for about 2 years. Surfline's cam is running perfectly and has been for at least a year. What's the hold up?

thermalben Friday, 25 Feb 2022 at 01:20 pm new

Now that we can travel more freely from Northern NSW, we've finally scheduled a large number of well-overdue upgrades (right around the country), which are happening over the coming weeks, This includes a major enhancement of our Sydney network. Very much looking forward to it.

thermalben Friday, 25 Feb 2022 at 05:17 pm new

Bloody hell. Primary surfcam server (in Amazon datacentre) went down around 3:30pm, and we couldn't revive it. Extremely unusual (i.e. haven't seen this in fifteen years). So, all surfcams are currently offline.

Lodged a critical support issue. Got a response 40 mins later: "Please be informed that System status check failures indicate that there is an issue with the hardware hosting. It is because of unhealthy underlying hardware the instance became unresponsive. And the instance must be migrated to a new, healthy host by stopping and starting the instance."

Righto, thanks.

And then an automated email another 30 mins later:

"Amazon has detected degradation of your hardware. Due to this degradation your server could already be unreachable. We will stop your instance after 2022-03-11 06:00:00 UTC. Please take appropriate action before this time."

So, there ya go - our internal monitoring picked it up before Amazon's automatic monitoring.

And now we begin the process to spin up a new server.

Apologies for the inconvenience.

Just what I need at 5pm on a Friday afternoon!

thermalben Friday, 25 Feb 2022 at 07:29 pm new

Server restored. Thanks for your patience.

Darksoda13 Monday, 14 Mar 2022 at 08:32 am new

thermalben wrote:

misterbrettc's comments above have got me thinking re: surfcam uptime. 

What is an acceptable period of downtime for a surfcam, over the course of a year? 15%, 10%, 5%, 1%?

If we were to aim for 95% uptime for each camera - i.e. 5% downtime - that would equate to a little over 18 offline days every year, almost three weeks. 

Obviously, there are some instances of downtime that are completely outside of our control, for example ISP network outages and power outages. But, for Swellnet hardware problems - where our equipment has broken down and needs replacing - I think there's merit in having a target.

We have our own internal monitoring system that keeps us notified as to when our surfcams go offline, and for a few years now I've been wanting to output the uptime information for each location. So, we might display something on the website like:

Snapper Rocks uptime

- 100% last day

- 100% last week

- 99.5% last month

- 97% last year

This would give subscribers some level of indication of the reliability of each camera over a period of time.

Because, in the case of Lorne and 13th Beach, although their current offline status is annoying, the fact is that both surfcams have been very reliable over the last ten (or more) years.

Darksoda13 Monday, 14 Mar 2022 at 08:52 am new

This isn’t because I don’t have faith in services like this, but because I know that services like this CAN and WILL AT TIMES become mostly defunct.

The same thing seems to be happening over and over again!?

Come on!! It’s mid March 2022 now...

mikehunt207 Sunday, 20 Mar 2022 at 12:16 pm new

Hi Ben
any info on the Margaret River cam? Been down for a while now.

thermalben Sunday, 20 Mar 2022 at 02:02 pm new

Yep, WA trip booked to come over and upgrade in a couple of week's time. Been waiting on a couple of things - borders to reopen, and also parts to become available. And then need to schedule that in amongst the other work we're doing.

Re: parts availability - this is another huge problem we've been battling for the last couple of years - there's a massive global shortage of chips required for a lot of the hardware we use, and it's creating huge delays in getting replacement equipment.

For example, we use two specific kinds of routers (depending on the job) and everywhere in the WORLD is out of stock at the moment, so I can't even order from US or EU. The next ETA for delivery in Australia is mid-July, so I'll receive goods mid-late July at the earliest - and even then, there'll be a massive backlog of orders - one of our suppliers has already sold 80% of their incoming stock - that requires payment up front (i.e. 4-5 months in advance), which is difficult for many small businesses to manage.

Anyway, the floods/rains have also caused a wide range of problems - we've lost quite a bit of hardware due to water ingress etc and are having to replace a lot of gear. Goes with the territory I suppose.

thermalben Monday, 4 Apr 2022 at 12:33 pm new

Wollongong back! Should be able to knock off most of the other offline cams this week and next. Thanks for your patience.

https://imgur.com/QN4QzNy.jpg

epictard Wednesday, 6 Apr 2022 at 11:19 am new

Nice to have it back and quality looks good. Cheers Ben.

surfing-cronulla Friday, 8 Apr 2022 at 10:48 am new

'Cronulla Point' [https://www.swellnet.com/surfcams/cronulla-point] is working OK but having problems. Noted above the cam had recently been replaced (and working fine) but for some reason the transmission stream is being affected somehow causing flickering and blotchy fogged out areas all over the place? Any ideas?
Checked on Google Chrome and Safari.

thermalben Saturday, 9 Apr 2022 at 06:28 am new

All of our Cronulla cams have been affected by the recent storms/rains (wide range of issues from power to water ingress), and although quite new - only a year old - will replaced ASAP (a large number of East Coast cams were similarly affected, and the team's been fixing them all up one by one). They're currently over in WA attending to surfcam upgrades there but will be back on the East Coast next week.

surfing-cronulla Saturday, 9 Apr 2022 at 03:13 pm new

thermalben wrote:

All of our Cronulla cams have been affected by the recent storms/rains (wide range of issues from power to water ingress), and although quite new - only a year old - will replaced ASAP (a large number of East Coast cams were similarly affected, and the team's been fixing them all up one by one). They're currently over in WA attending to surfcam upgrades there but will be back on the East Coast next week.

OK thanks for the heads up. Thought maybe something like this had happened. One hell of a workload ahead.

thermalben Sunday, 10 Apr 2022 at 09:55 am new

OK.. Margs cam is back!

But.. Cronulla cams (beaches and Alley) are still offline, and it looks like Shark Island finally died this morning (it's been having issues ever since the huge rain event a few weeks back).

I actually ordered a cam upgrade for Shark Island a couple of weeks ago, but the hardware is still sitting in a container in China - delayed once again due to Covid (distributor says: "Shanghai is in very bad Covid situation now. They are locking down for the warehouse for the moment even Shanghai airport, we just have to wait for they are back.") So, unsure how long that will be. Currently taking 3-4 months for hardware to arrive in Australia, once it's been paid for.

Anyway, we have some other gear in the office that'll work in the meantime, so we'll try to get down to the Shire this week to swap 'em all out.

We've had a couple of really tricky weeks thanks to the adverse weather, and are also experiencing across multiple other sites (ongoing NBN issues at Yamba, dead surfcam Greenmount etc) - but we'll get to 'em all ASAP. Got a couple of months of work ahead including a large number of new surfcam installs too.

Thanks for your patience.

Supafreak Sunday, 10 Apr 2022 at 09:57 am new

Geez malben you’re having a bad run , when it rains it pours.

thermalben Sunday, 10 Apr 2022 at 10:42 am new

Thing that disappoints me is that is masks the amazing work we've done behind the scenes across the entire cam network.

Might have to do an explanatory article soon highlighting what we've built.

mikehunt207 Sunday, 10 Apr 2022 at 01:30 pm new

Nice to see the Margaret River webcam is back finally, been working away from the coast so good to be able to see conditions again , One slight correction needed; the boat ramp or boaties but boat ramps? SN renaming surfspots? Its only ever been since Kelly slater and the wsl tried to land grab more spots for their contest did " boat ramps" become terminology (you can tell how long someone has been surfing around here if they call it boat ramps - not long)
While on topic , the wsl and their surfing wa minions have taken over the carpark at Margs with tents and fences, 3 weeks before they even begin (Bells hasnt even started yet) , school holidays, easter and ANZAC day and the point locked up to the locals and tourists , what a fucking joke our Shire is , dodgy permits for multiple contests all summer including SUP, Kite surfing and junior contests, locals wonder how come so many new comps can get approved and then we see Shire council member also has contract to do all the electrical work on contest sites , suss as fuck
fuck the wsl

thermalben Sunday, 10 Apr 2022 at 01:45 pm new

Fair point MH. I actually thought it was Boatramp (one word) but as per usual did a quick Google and seemed to come across more cases of the pluralised, two-word version.

I'll amend. One word or two?

mikehunt207 Sunday, 10 Apr 2022 at 02:00 pm new

Boat Ramp or Boatramp are both fine (there is only one ramp there not multiple ha ha )
thanks for that, now if you can do something about the crowds out there these days....

Michael Adam Sunday, 10 Apr 2022 at 03:14 pm new

Also a note to new residents, VALs and visitors: it’s Margs or Margarets, never Margies or Margis.
Also: Bali is now OPEN!!!!
Thank you and goodnight.

thermalben Sunday, 10 Apr 2022 at 04:13 pm new

Yeah the Margies moniker has bugged me for years.

thermalben Thursday, 14 Apr 2022 at 09:24 am new

Experiencing some issues with our Burleigh data connection this morning, have sent a tech around to investigate.

Sprout Tuesday, 26 Apr 2022 at 10:58 am new

Shark Island ETA?

Join the conversation Sign up for free OR Login