Warning: file_get_contents(/home/www/settings/mirror_forum_db_enable_sql): failed to open stream: No such file or directory in /var/www/html/content/Forum/functions.php on line 8
Here's the thing though: William is a former radio DJ. He knows exactly how bad "dead air" is. You get a massive drop off in listeners, and it takes a while to recover. That translates into lost revenue.
When a commercial radio stations had to do planned maintenance to backend infrastructure, such as repairing or replacing a rack of audio equipment, they plan ahead with an alternate broadcast to avoid complete radio silence during the actual maintenance window going on behind the scenes. typically this would entail playing a long tape with pre-recorded music.
Alanna is framing this outage as cleaning up a house that's been lived in for 26 years. This definitely implies that the problems were known, and that some kind of maintenance window could've been created.. Jarred could have altered the API to play the same music on a loop for 24 hours along with a recurring announcement recorded by William or Alanna or Josh that certain functions would be unavailable (downloads, comments, voting, etc).
That none of these contingencies were planned or prepared, and that the maintenance went ahead means that there is a critical lack of planning and redundancy happening at Radio Paradise.
I'm a career IT professional; if I made a change like this without announcing it, I'd be seriously reprimanded. Possibly fired.
William is playing a bumper on the main mix where he touts that donations from loyal listeners have been able to find employment for 12 staff members who never have to look for another "real job" because working at Radio Paradise is amazing.
I know it's easy for me to critique the station from here... but I've actually been to the station at Eureka. On the surface, it looks to be a very well-oiled machine and a squeaky-clean operation. And it is abundantly clear that all the staff love the station and want it to keep going forever. Yet... out of those 12 people, did anyone raise ANY alarms or red flags?
This organization is too mature to be making "aw shucks" mistakes. The day long dead-air could translate into lost donation revenue.
My thoughts exactly. As a user and code contributor to the Lyrion Music Server open source project, I am very disappointed in the seemingly haphazard way that the RP API has repeatedly been broken, fixed, broken again, changed, and otherwise bounced around for the past month or so. This is not housecleaning. It is uncontrolled chaos. Many Lyrion users, including myself, are becoming very frustrated and I have no doubt that many are just moving on to other more stable, if less excellently curated, streaming sources. Please try to do better. We are all rooting for you.
I think that an in-depth "post-mortem" report would go a long way to helping the community understand what caused the outage, and gain a deeper appreciation of the complexity of RP, and whatever was being cleaned up under the hood.
In case anyone's resting on their laurels thinking it's all good now, I'm still getting disruptions. It happened on both the iOS app and Lyrion Music Server. Here's the error that the RP plugin for LMS just threw:
Plugins::RadioParadise::ProtocolHandler::new (29) We seem to be in a redirection loop for url: https://audio-geo.radioparadise.com/dj/4/10014873.flac
Edit: I will say that things are slooowly getting better, so it's at least going in the right direction. Trying not to sound like a whiny little bitch.
I'm glad to hear that this was a planned outage. I suffered a bit for a day but almost everything seems to be back to normal today (website, Mac OS App, and IOS App) with one major exception. On the IOS App, the download still produced the following error when attempting to play the downloaded file - "Failed to start local playback. Please try again." Multiple retries does not help unfortunately. Any ideas why this is happening or when it will be fixed?
@dryan67, please take care not to confuse planned maintenance with a planned outage.
When properly documented and rehearsed, plannedmaintenance can be done on live systems without causing a noticeable or extended disruption.
A plannedoutage, on the other hand, means: "we have run all reasonable scenarios for this work, and there's no way we can do it without disrupting the live service for more than a few seconds, therefore we are planning to take it completely offline as part of the work."
This was planned maintenance with an unplanned outage.
Despite what @lovehorn wrote (and later deleted), I am not "emotion-express on steroids" and I don't think my comments can be construed as "swearing." These are the frustrations of a 20-year career IT systems administrator who has done his fair share of planned and unplanned maintenance, as well as surviving the stress of both planned and unplanned outages.
In my line of work, I am always expected to have a "rollback plan," unless the work is a "one way" fix that can't be rolled back. This means, if an outage occurs, there needs to be a way to revert to the previous settings to restore service as quickly as possible. Computer code doesn't burn up or vaporize like real hardware. That's what backups are for.
I have also been listening long enough to see RP and its sister station, SomaFM, go through some outages before. Rusty, for his part, seems to do pretty good learning lessons from unplanned outages.
When RP posted an IT job 2-3 years ago, I really wish I could have just up and left Los Angeles and come to work for them instead. But I wasn't in a position to just leave the area and move my entire life up north.
So instead, I am simply pleading for more careful planning in the future so that everyone can enjoy the service for years to comeâlong after William has spun his last record.
I'm glad to hear that this was a planned outage. I suffered a bit for a day but almost everything seems to be back to normal today (website, Mac OS App, and IOS App) with one major exception. On the IOS App, the download still produced the following error when attempting to play the downloaded file - "Failed to start local playback. Please try again." Multiple retries does not help unfortunately. Any ideas why this is happening or when it will be fixed?
I'm glad to hear that this was a planned outage. I suffered a bit for a day but almost everything seems to be back to normal today (website, Mac OS App, and IOS App) with one major exception. On the IOS App, the download still produced the following error when attempting to play the downloaded file - "Failed to start local playback. Please try again." Multiple retries does not help unfortunately. Any ideas why this is happening or when it will be fixed?
This is a software system set up by an IT self educated DJ
without a necessary implementation of any of the industry-standards you mention.
Plus unexpected growth.
Plus ...
Be nice!
Be thankful!
I am a recurring donor for several years now. That is how I express my thanks.
I am also allowed to express my frustration. William ostensibly hired all these people to help out because the system has reached some growth limit.
People who donate to nonprofit organizations expect them to be good stewards with their funds and do everything they can to keep the organization healthy and strong.
That includes not jeopardizing future revenue streams.
Here's the thing though: William is a former radio DJ. He knows exactly how bad "dead air" is. You get a massive drop off in listeners, and it takes a while to recover. That translates into lost revenue.
When a commercial radio stations had to do planned maintenance to backend infrastructure, such as repairing or replacing a rack of audio equipment, they plan ahead with an alternate broadcast to avoid complete radio silence during the actual maintenance window going on behind the scenes. typically this would entail playing a long tape with pre-recorded music.
Alanna is framing this outage as cleaning up a house that's been lived in for 26 years. This definitely implies that the problems were known, and that some kind of maintenance window could've been created.. Jarred could have altered the API to play the same music on a loop for 24 hours along with a recurring announcement recorded by William or Alanna or Josh that certain functions would be unavailable (downloads, comments, voting, etc).
That none of these contingencies were planned or prepared, and that the maintenance went ahead means that there is a critical lack of planning and redundancy happening at Radio Paradise.
William is playing a bumper on the main mix where he touts that donations from loyal listeners have been able to find employment for 12 staff members who never have to look for another "real job" because working at Radio Paradise is amazing.
I know it's easy for me to critique the station from here... but I've actually been to the station at Eureka. On the surface, it looks to be a very well-oiled machine and a squeaky-clean operation. And it is abundantly clear that all the staff love the station and want it to keep going forever. Yet... out of those 12 people, did anyone raise ANY alarms or red flags?
This organization is too mature to be making "aw shucks" mistakes. The day long dead-air could translate into lost donation revenue.
What you're missing:
This is a software system set up by an IT self educated DJ
without a necessary implementation of any of the industry-standards you mention.
To be clear, my post was IN NO WAY trying to say that anyone should be thinking about taking the keys away from anyone, and I want to disavow any and all connections between my post and any such idea.
I just want a little more communication, that's all.
Here's the thing though: William is a former radio DJ. He knows exactly how bad "dead air" is. You get a massive drop off in listeners, and it takes a while to recover. That translates into lost revenue.
When a commercial radio stations had to do planned maintenance to backend infrastructure, such as repairing or replacing a rack of audio equipment, they plan ahead with an alternate broadcast to avoid complete radio silence during the actual maintenance window going on behind the scenes. typically this would entail playing a long tape with pre-recorded music.
Alanna is framing this outage as cleaning up a house that's been lived in for 26 years. This definitely implies that the problems were known, and that some kind of maintenance window could've been created.. Jarred could have altered the API to play the same music on a loop for 24 hours along with a recurring announcement recorded by William or Alanna or Josh that certain functions would be unavailable (downloads, comments, voting, etc).
That none of these contingencies were planned or prepared, and that the maintenance went ahead means that there is a critical lack of planning and redundancy happening at Radio Paradise.
I'm a career IT professional; if I made a change like this without announcing it, I'd be seriously reprimanded. Possibly fired.
William is playing a bumper on the main mix where he touts that donations from loyal listeners have been able to find employment for 12 staff members who never have to look for another "real job" because working at Radio Paradise is amazing.
I know it's easy for me to critique the station from here... but I've actually been to the station at Eureka. On the surface, it looks to be a very well-oiled machine and a squeaky-clean operation. And it is abundantly clear that all the staff love the station and want it to keep going forever. Yet... out of those 12 people, did anyone raise ANY alarms or red flags?
This organization is too mature to be making "aw shucks" mistakes. The day long dead-air could translate into lost donation revenue.
To be clear, my post was IN NO WAY trying to say that anyone should be thinking about taking the keys away from anyone, and I want to disavow any and all connections between my post and any such idea.
I just want a little more communication, that's all.
A MEA CULPA was uttered here before by the KING himself for shortcomings in communication by William of the Gold Mountain Sire.
His Lordship rather focusing on fixing errors in a 1-2 man band, rather than communicating them.
Planned maintenance and outages are something you communicate to your users and the public well in advance. This is not RP's first rodeo with maintenance windows.
Play an announcement multiple times a day on all stations. Alter the metadata feed so instead of the actual artist, song title and album it says PLANNED OUTAGE / STATIONS OFFLINE / MM-DD-YY @ HH:MM.
If William is still content to monkey around inside the machine and has been hand-waving concerns from his staff about possible outages, and staff are acquiescing because he's the owner and founder... then maybe it's time for his daughter to take this as a serious lesson learned and have a conversation about "taking the keys away."
To be clear, my post was IN NO WAY trying to say that anyone should be thinking about taking the keys away from anyone, and I want to disavow any and all connections between my post and any such idea.
I just want a little more communication, that's all.
Hi everyone â
Just wanted to let you know weâve been doing some fairly heavy lifting behind the scenes on our API, servers and database. Itâs the kind of deep-clean, infrastructure work that isnât very glamorous⦠but very necessary.
If you imagine RP as a house â and we turned 26 this year â weâve accumulated a fair amount of unnecessary stuff over time. And as anyone whoâs done a serious clean-out knows, when you start moving boxes around, sometimes things temporarily go missing.
Unfortunately, thatâs led to a few unexpected (and sometimes mysterious) outages. To be honest, more than we are comfortable with. We know how frustrating this is on your end.
Thereâs never really a âgoodâ time to tackle this kind of maintenance â but itâs important work that will make RP stronger and more stable in the long run.
The good news: William and Jarred have been incredibly quick to jump in and get things back on track each time something hiccups.
Thank you for your patience while we tidy up the engine room. Itâs all in service of a better Radio Paradise. We promise.
...This is not RP's first rodeo with maintenance windows.
... "taking the keys away."
Messing with this complex music and listener-commentary database on a live system(!) would be named as suicide by any IT business, regardless of their field of expertise.
Barbers and hotdog-cooks usually know their business without any database schema because they don't exactly need it.
They got what they need.
RP has continued to be a One-Man-Show like Fürst Bismarck
Planned maintenance and outages are something you communicate to your users and the public well in advance. This is not RP's first rodeo with maintenance windows.
Play an announcement multiple times a day on all stations. Alter the metadata feed so instead of the actual artist, song title and album it says PLANNED OUTAGE / STATIONS OFFLINE / MM-DD-YY @ HH:MM.
If William is still content to monkey around inside the machine and has been hand-waving concerns from his staff about possible outages, and staff are acquiescing because he's the owner and founder... then maybe it's time for his daughter to take this as a serious lesson learned and have a conversation about "taking the keys away."
He always used to announce when he was going to tinker with stuff, but it's exponentially more complicated now and they did have some rollouts that went off without a hitch.
I second this. If this was planned a heads up would have been much appreciated, and if it wasn't planned, someone, anyone, coming into this thread and letting us know that you're aware of the issues and are working on it would also be appreciated.
Seconded, thirded, and fourthed.
Planned maintenance and outages are something you communicate to your users and the public well in advance. This is not RP's first rodeo with maintenance windows.
Play an announcement multiple times a day on all stations. Alter the metadata feed so instead of the actual artist, song title and album it says PLANNED OUTAGE / STATIONS OFFLINE / MM-DD-YY @ HH:MM.
If William is still content to monkey around inside the machine and has been hand-waving concerns from his staff about possible outages, and staff are acquiescing because he's the owner and founder... then maybe it's time for his daughter to take this as a serious lesson learned and have a conversation about "taking the keys away."
I can sure understand that. I want RP to last forever, so a little house cleanup makes sense. Perhaps just let us know in advance so we aren't questioning everything. And tell your API thanks for the lifelong soundtrack at an amazing price.
I second this. If this was planned a heads up would have been much appreciated, and if it wasn't planned, someone, anyone, coming into this thread and letting us know that you're aware of the issues and are working on it would also be appreciated.
Hi everyone â
Just wanted to let you know weâve been doing some fairly heavy lifting behind the scenes on our API, servers and database. Itâs the kind of deep-clean, infrastructure work that isnât very glamorous⦠but very necessary.
If you imagine RP as a house â and we turned 26 this year â weâve accumulated a fair amount of unnecessary stuff over time. And as anyone whoâs done a serious clean-out knows, when you start moving boxes around, sometimes things temporarily go missing.
Unfortunately, thatâs led to a few unexpected (and sometimes mysterious) outages. To be honest, more than we are comfortable with. We know how frustrating this is on your end.
Thereâs never really a âgoodâ time to tackle this kind of maintenance â but itâs important work that will make RP stronger and more stable in the long run.
The good news: William and Jarred have been incredibly quick to jump in and get things back on track each time something hiccups.
Thank you for your patience while we tidy up the engine room. Itâs all in service of a better Radio Paradise. We promise.
I can sure understand that. I want RP to last forever, so a little house cleanup makes sense. Perhaps just let us know in advance so we aren't questioning everything. And tell your API thanks for the lifelong soundtrack at an amazing price.
Hi everyone â
Just wanted to let you know weâve been doing some fairly heavy lifting behind the scenes on our API, servers and database. Itâs the kind of deep-clean, infrastructure work that isnât very glamorous⦠but very necessary.
If you imagine RP as a house â and we turned 26 this year â weâve accumulated a fair amount of unnecessary stuff over time. And as anyone whoâs done a serious clean-out knows, when you start moving boxes around, sometimes things temporarily go missing.
Unfortunately, thatâs led to a few unexpected (and sometimes mysterious) outages. To be honest, more than we are comfortable with. We know how frustrating this is on your end.
Thereâs never really a âgoodâ time to tackle this kind of maintenance â but itâs important work that will make RP stronger and more stable in the long run.
The good news: William and Jarred have been incredibly quick to jump in and get things back on track each time something hiccups.
Thank you for your patience while we tidy up the engine room. Itâs all in service of a better Radio Paradise. We promise.
