How to "Learn Stan" by Osmosis

General
#1

I’m debating a couple of undergraduate courses for the winter. Basically I can take either MATH 2505 Introductory Analysis or MATH 3210 Intro to Numerical Analysis not both and based on my subsequent two years I probably won’t be able to return to do the one I pass on. 2505 is an Honors class (iz fine) and a prerequisite for some serious Analysis classes, yet I am planning on focusing on Abstract Algebra (perhaps in some ways it’s more relevant to doing CS research) and wouldn’t necessarily take advanced Analysis on personal interest alone.

Another stream of courses I take leads me through stat theory to fourth year Bayesian Data Analysis, also my major is CS so I get data structures and algorithms, but curiously we don’t have formal numerical analysis.

Here are some course definitions:

2505 - “Axioms for the real number system, geometry and topology of Euclidean space, limits, continuity, differentiability, the inverse and implicit function theorems.”
|_ 3501 - “Metric spaces, point-set topological notions, sequences, completeness, separability, compactness (Heine-Borel, Bolzano-Weierstrass, Finite Intersection, complete and totally bounded), limits and continuity, continuity in topological terms, connectedness, path and local-path connectedness, homeomorphisms, uniform continuity, Lipschitz continuity, contractions, contraction principle, sequences of functions, uniform convergence.”
|_ 3502 - “The full derivative for functions between Euclidean spaces, directional derivatives, Jacobian matrix, differentiability, C¹ functions, multilinear maps, higher derivatives, Taylor’s theorem, extrema, inverse and implicit function theorems, extrema subject to constraints, Lagrange multipliers.”

3210 - “Root finding, interpolation, integration, initial value problems, linear and nonlinear fitting, boundary value problems, error analysis and stability of methods, as well as practical computer implementation.”

My interest is to research and contribute new functions into the Stan math library, which course of the two would you recommend as more rewarding (even indirectly)? Is it a mistake in the Stan/BDA world to avoid Analysis in favor of Algebra? Which course would you take?

I’m really totally split on the decision.

#2

If you’re motivated you can learn anything you want on your own.

I like math. Analysis would be a really good idea, and it’s a fun class, and it comes in handy. Numerical analysis is fun, but it’s more relevant to applied mathematics than prob. machine learning. The numerical stability stuff is helpful though, and some fun proofs related to it.

A lot of this stuff is special topics anyway. Like MCMC? I learned about it as a special topics Stats 414 from Atchade. We implemented adaptive metropolis algorithms and the like in R. I can send you the notes if you want, and like, an adaptive metropolis or something, but they’re not really scalable algorithms. But even in Bayesian stats classes (at least here at Michigan) they’re teaching Bayesian inference with like, BUGS, and Gibbs samplers aren’t really scalable, as I’ve painfully learned from experience, implementing bad samplers on large models with conjugate priors.

I think this stuff takes a lot of conscious effort, and osmosis I’ve noticed works better with stuff like language (aural and verbal) and music, not so much with mathematics…

1 Like
#3

In my experience I’ve met people on the CS track who never quite understand why statistics is not just math so I’d say don’t expect to understand Bayesian data analysis if you don’t spend some time on it directly. That’s pretty important for contributing to the development of applied Bayesian analysis. OTOH if you just want to contribute to the machinery then either way is fine, all the math you mention shows up somewhere in the machinery and would let you contribute… abstract algebra is the only one I’m not sure about but my math background is not formal so somebody else could say more confidently.

1 Like
#4

The ACM offers categorization of these topics

delivery.acm.org/10.1145/2380000/2371137/ACMCCSTaxonomy.html

So functional analysis is part of the “Mathematics of Computing” class and is a category consisting of only on topic: approximation. Perhaps the smallest category of them all! The numerical analysis category is in this same class with many individual topics.

The Abstract Algebra topics (maybe not truly the “Abstract” part - that’s just the name of the class) are at the top of the “Computing Methodologies” class.

Also I remembered reading this post a few months ago which got me to consider a class in numerical analysis in the first place: https://andrewgelman.com/2018/07/12/learn-log_sum_exp-log1p-lccdf-numerical-analysis-tricks/

#5

I don’t follow the newest machine learning literature too closely, but I do always see a lot of optimization papers at places like NIPS. Optimization is a subfield of numerical analysis.

1 Like
#6

Yeah you are correct, listen to Arya!

1 Like