Bayesian

[RogerS]

I saw reference to the existence of this word following the tragic death of Mike Lynch and friends. I had vaguely heard of the term but didn't really know what it was all about. So I Googled it

Bayesian optimization is a sequential design strategy for global optimization of black-box functions that does not assume any functional forms. It is usually employed to optimize expensive-to-evaluate functions

I still don't know.

 

[AJB Temple]

Probability, but with belief and new data added to existing data. Which makes it conditional probability.

It's ironic really, given that everyone involved with that boat would consider that the probability of a catastrophic downdraught hitting a big yacht during a forecast force 5 (fresh breeze of minor magnitude) must be quite small, and the chances of it sinking smaller still. Very sad.

 

[Mike G]

Bayesian logic is how matey made his fortune. He was a mathematician originally, and simply applied his knowledge of this form of probability to a number of computer programmes.

 

[RogerS]

An initial delve into Autonomy and Bayesian techniques looks fascinating.

 

[Tiresias]

My partner uses it in malaria research. And my father used it in his PhD thesis (a real page turner) many moons ago. It’s been in and out of fashion.

There have been attempts to use it in expert systems for property valuation, but as far as I know it has come to naught.

 

[Dr.Al]

Back in the day, I seem to remember Bayesian techniques being the method of choice for spam detection on email servers. That's about the limit of what I know about it though.

 

[RogerS]

Back in the day, I seem to remember Bayesian techniques being the method of choice for spam detection on email servers. That's about the limit of what I know about it though.

Ah thank you for that. I knew that I'd heard it before and you've just supplied the missing piece.

 

[SamQ aka Ah! Q!]

"Bolean logic and Bayesian techniques". Title of an introduction to digital search techniques by my university library, back in 1979. Pre Google. I paid £37 for 0.12 seconds of computer run time on (I think) "George", Manchester University's three floor, mainframe computer.

 

[Tiresias]

"Bolean logic and Bayesian techniques". Title of an introduction to digital search techniques by my university library, back in 1979. Pre Google. I paid £37 for 0.12 seconds of computer run time on (I think) "George", Manchester University's three floor, mainframe computer.

Christ. The programmes my father used to run for his magnum opus were run all the way through the night. In Edinburgh University in the late 70s or at the APRE. I’d have to ask who paid for it – either the University or the MoD. I do know he did get asked why he needed so much computing time.

My partner does run some analysis programmes overnight, but with quite (understatement) a lot more data. And not on a mainframe.

 

[Xlurker]

The simplest explanation of Bayesian statistics that I have found is in the link below.

It is a good read even if you skip the sums because it gives examples of the horrendous consequences of not using Bayesian calculations when you should.
For example an innocent person being convicted of a double murder.

The obscure maths theorem that governs the reliability of Covid testing

There’s been much debate about lateral flow tests – their accuracy depends on context and the theories of a 18th-century cleric

www.theguardian.com

 

[SamQ aka Ah! Q!]

Christ. The programmes my father used to run for his magnum opus were run all the way through the night. In Edinburgh University in the late 70s or at the APRE. I’d have to ask who paid for it – either the University or the MoD. I do know he did get asked why he needed so much computing time.

My partner does run some analysis programmes overnight, but with quite (understatement) a lot more data. And not on a mainframe.

Tiresias? I was simply making two points: some (particularly statistical) operations seem to come and go cyclicly, and secondly, we have come a long way in execution of same.
1980, me old mate Steve, looking at circadian rhythm modification by tides (and moon) in hermit crabs, entered 3,500 date points - by hand!! - into a PDP8 computer for a Fourier analysis. It was 47 minutes before it typed the first line of stats.
I had an ex-pupil demonstrate much the same about 7 years sgo, just before I retired. The data went straight in from sensors and was available on screen (and for printing) instantaneously. Times change.

 

[RogerS]

Latest support 'newsletter' from SpamSieve has this exchange which, as I now realise, is a perfect example of the impact of prior odds.

Customer - Aug 20

After 10 years of use and careful training SpamSieve has a 98.6 accuracy rate still. But I think I have found its “spam kryptonite”. It appears that grammatical English, standard phrasing and conventionally formatted emails offering my business “credit, lines of credit, financing, or loans” all get through with a score of 27 (or 5, like this one) and are almost always “Predicted Good”.
I have trained a few dozen of these but one or two still gets through every week. Not a big problem just an observation. I did think that repeated training would somehow reveal a pattern in these solicitation emails that annoy me though.

Support - Aug 20

That’s a low accuracy rate for SpamSieve in general, and if you’ve been training for 10 years there’s likely a lot of information that’s now out-of-date. It would probably help to choose Reset Corpus from the File menu, then re-train SpamSieve with a smaller number of recent messages.


Customer - Aug 20

It’s been right about 98.6% for years. No doubt old stuff in corpus, but I guess I can retrain. I will need to save up some spam for a few days to get to the right training level percentage of 65/35 as I tend to trash spam fast, and had been set to auto-delete any spam I train manually.

Customer - Aug 22

Well I took the plunge. I had treated my 10-year old corpus like delicate sourdough starter and the thought of resetting/discarding it was scary. I did. Retrained with 65 spam and 35 good messages and I am on my way again! New stats show 99.1% accuracy 3.

 

[Tiresias]

Tiresias? I was simply making two points: some (particularly statistical) operations seem to come and go cyclicly, and secondly, we have come a long way in execution of same.
1980, me old mate Steve, looking at circadian rhythm modification by tides (and moon) in hermit crabs, entered 3,500 date points - by hand!! - into a PDP8 computer for a Fourier analysis. It was 47 minutes before it typed the first line of stats.
I had an ex-pupil demonstrate much the same about 7 years sgo, just before I retired. The data went straight in from sensors and was available on screen (and for printing) instantaneously. Times change.

Sam. I think we’re making the same points. Fashions change in statistical analysis; Bayesian stuff needs a shed load of computing power; we have a lot more computing power than ever could have been imagined in the late 70s.

My father’s data set for his PhD was essentially every one that had ever been in the British military forces. And you know how the MoD like keeping records. Even if they are fictitious. All entered by hand.

My partner’s data analysis is even more overwhelming. But they just have an old desktop they leave to churn out stuff. It’ll probably become self-aware at some point. And conclude that the best way of eradicating malaria is human extinction.

Incidentally, when were you at Manchester University? My father would have been there in the 60s. Or were you just using their mainframe?

 

[Lurker]

Never ceases to amaze me, the depth and breadth of knowledge of fellow forum members.

 

[AndyP]

1980, me old mate Steve, looking at circadian rhythm modification by tides (and moon) in hermit crabs

Sorry Sam I have to ask. Why?

 

[SamQ aka Ah! Q!]

Sam. I think we’re making the same points. Fashions change in statistical analysis; Bayesian stuff needs a shed..........

Incidentally, when were you at Manchester University? My father would have been there in the 60s. Or were you just using their mainframe?

Nah. Just digitally pasting through.

I was at uni 1977-81, so no chance of overlapping.

I was a T.A..ossifer for 27 years though, know exactly what you mean re army recording practices! What was the form....762? That you signed everything out on?

 

[SamQ aka Ah! Q!]

Sorry Sam I have to ask. Why?

Stephen was a computer junkie. This was an opportunity - once the hard work of data gathering was done, to play at the cutting edge of data analysis, at a time when we had been using slide rules just 3 years previously at A level.

He was also deeply intrigued by the variables influencing seasonal and temporal behaviours. A visiting lecturer had spiked his interest in this field and Stephen spotted an opportunity to work in an area away from the mainline foci of most of our faculty. That gamble paid off with a professorship some years later.

 

[wallace]

My brain hurts just attempting to understand

 

[AJB Temple]

I was sponsored by a firm to do my PhD in the mathematics involved in the creation and perpetual mark to market valuation of derivative instruments when I was in my mid 20s (quite some time ago ) and the university billed them what I considered to be a lot each week for the computing time that was used. They also insisted on a junior member of the staff supervising the running of the computer for overnight processes, and billed for that time too although there was in reality nothing to see and nothing to supervise. It seems weird now looking back on it. You actually had to book out computer time and sign for it. I suspect my phone today is more powerful than the computers we were using back then. It is quite amazing to me still how far technology has come on in our lifetimes.

 

[Phil Pascoe]

It used to be said that a 7 series BMW had more computing power than the last spaceship that went to the moon - now it's a singing greetings card.