Hey, I'm Sebi. I'm writing my master's thesis on Reinforcement Learning and decided to document the whole mess publicly: progress, dead ends, concepts I'm tr...
After finishing the "Background" part I am now focusing on the "Related Work" Chapter. Again I thought I might share the structure and rough first draft. Background "[...] Atari is employed in this thesis not as a subject of study, but as a controllable testbed in which appearance, dynamics, and rewards can be independently modified." Clean transition into related work. Related Work Atari…
I'm currently working on the background chapter for the thesis and wanted to share the structure and rough first draft of it here. My governing principles: The background should contain exactly what the reader needs to understand the central argument and methology and should be clearly separated from related work. It should lay the conceptual foundation (i.e. definitions, formalism, mechanisms)…
Benchmarking CRL under controlled distribution shift with JAXtari (working title) CRL agents must learn sequences of tasks without forgetting what they already know (catastrophic forgetting). Evaluating how well they do this turns out to be hard to do rigorously. The dominant approach (in the realm of Atari games for RL benchmarking) trains agents across different games in sequence. The problem is…
Lecture 2: Markov Decision Processes An agent learns by interacting with its "environment". How can we formalize this? MDPs Sequence of discrete time steps t = 0,1,2,... At each step Receive representation of state S t Execute action A t Obtain reward R t and reach a new state S t + 1 Sequential interaction between agent and environment generates a trajectory : S 0 , A 0 , R 1 , S 1 , A 1 , R 2 ,…
Why? I want to build my RL foundations in a more systematic way both by reading literatuer and visiting the RL course at TU Darmstadt this summer term. For this purpose it's nice that I am completely done with my course work so I can focus on the thesis. This gives me the freedom to visit interesting lectures where necessary and skip the ones that I don't want to hear. Reading List Reinforcement…
I was looking for some papers on how to quantify the difficulty in RL environments as well as if there is work on how humans perceive difficulty compared to RL agents. Here is my short, curated list on this (I did not read those in detail yet since I have other stuff to attend to, but I will do in the future): Hardness in Markov Decision Processes: Theory and Practice Surveys hardness notions for…
Reading List While I found a lot of material to read, I will try to keep it as lean as possible to start off. I also need to refresh and/or broaden my knowledge on RL in general (since i think its a good starting point to assume that I know little or that what I know is probably wrong). But at least for today thats not the point. However I will start to re-visit basic RL topics in a more…
Before getting all excited and diving deeper into my idea of a forgetting benchmark, I want to run some sanity check (plus my supervisors think it's a good idea). These initial experiments will provide me with some orientation, traction and, hopefully, guide me in a clearer direction. I sat down and wrote a short abstract on what I plan to do. An abstract (kind of) We conduct an initial…
When I was starting to look for where to write my thesis I took some notes in a physical notebook (which I will likely continue to do), and I thought I might just "digitalize" them here. Or rather jot them down to give them some structure and maybe gain some insight. Idea 1 I'll cut the other fields I was considering and only focus on what I wrote down about RL. First thing I asked myself was how…
Kickoff for my thesis So the first thing I need to do is find a topic to research, build and write about. And while it may sound weird this means finding a problem to solve (and ideally a problem to fall in love with) because only then will I be motivated and only then will the work be at least a bit meaningful/useful. Since I already worked at a lab that does a lot of research on RL (at TU…